Real-time Facial Expression Recognition “In The Wild” by Disentangling 3D Expression from Identity

This section describes how we process a very large-scale dataset to construct pseudo-ground truth, which was used to train the DeepExp3D, a robust CNN capable of regressing the 3DMM expression parameters from a single RGB image. We start from a collection of 6,000 RGB videos with 12 million frames in total and 1,700 unique identities. Please refer to the Supplementary Material for the specifics of the collection process. In every frame of every video of our video collection, we applied the method of [12] to detect faces and extract from each detected face a set of 68 landmarks, according to the MULTI-PIE markup scheme [29]. Afterwards, we applied the following steps:
False detections removal: This was implemented by tracking each detected face in the first frame throughout the processed video. A face is kept if its bounding box (BB) stays within a reasonable margin, chosen experimentally to be half the width of the BB, compared to its location in the previous frame. We pruned videos in which we lost track of the face for $K$ consecutive frames (chosen experimentally to be 5) before reaching the desired number of tracked frames $F$ (chosen experimentally to be 2000). This step helped to remove false detections arising due to a failure in the face detector or out-of-context detections, e.g. a facial photo in the background of a video, faces that pop in/out of the camera viewing angle, etc. This step resulted in pruning 1000 videos (16.7% of the initial dataset).
Temporal smoothing: Extracted landmarks were temporally smoothed using cubic splines. This was performed to alleviate the effects of the potential jitters in the extracted landmarks between consecutive frames and to fill in the possible gaps (frames with lost tracking) that persisted for less than $K$ frames.
3D facial reconstruction from videos: For every video, we followed the process described in Sec. III-A and estimated the facial shape parameters ($\mathbf{i}$, $\mathbf{e}_{f}$ for $f=1,...,F$). The final output of pseudo-ground truth creation is the sequence of expression vectors $\{\mathbf{e}_{f}\}$. However, we also utilise the identity vector $\mathbf{i}$ in the next step as one means of error pruning.
Error pruning: With such a large number of videos, there will be some cases of videos where 3D reconstruction has failed. This is an unavoidable byproduct of the fact that the adopted landmark localization, even though very robust, might not be sufficiently accurate for cases of extremely challenging facial videos. Our approach compensates for that by two stages of pruning problematic videos:

a) Automatic pruning: We are based on the fact that under the adopted 3D face modelling (1), the coordinates of the estimated identity vector $\mathbf{i}$ of each video are assumed independent, identically distributed random variables that follow a normal distribution. Therefore, we classify as outliers and automatically prune the videos that correspond to an estimated value of $\|\mathbf{i}\|$ above an appropriate threshold. More details are given in the Supplementary Material. This resulted in automatically pruning 300 more videos (5% of the initial dataset).

b) Manual pruning: There might be a few problematic videos that “survived” the automatic pruning. For that reason, we inspected the reconstructions of all remaining videos and manually flag and prune videos where it is evident that the 3D face reconstruction has failed. In this step we manually pruned 250 videos (around 4% of the initial dataset).

To conclude, our constructed training set consists of videos of our collection that survived the aforementioned steps of video pruning. It consists of 5,000 videos (83.33% of the initial dataset) with 1,500 different identities and around 9M frames. For exemplar visualisations, please refer to the Supplementary Material.

The constructed training dataset of videos is rich in the facial expressions that are viewed from different angles and under various illumination conditions throughout the video, as well as in identities (1500 in total). This substantially facilitates the process of training a Convolutional Neural Network (CNN) $\mathcal{N}:I\rightarrow\mathbf{e},$ aiming at regressing the 3DMM facial expression coefficients (referred to as $\mathbf{e}$ in equation (1)) from a given RGB image $I$. The network $\mathcal{N}(I)$ learns during the training phase how to map from the image space to the facial expression space irrespective of the subject’s identity shown in image $I$. This is achievable by the virtue of the utilized facial 3DMM which represents the reconstructed face as a summation of identity and expression parts on top of the model mean face ($\mathbf{\bar{x}}$), see equation (1). We extract vectors $\mathbf{e}$ from our dataset as a result of the fitting approach explained in section III-A and use them as pseudo annotations for training $\mathcal{N}$ in a supervised manner. However, to avoid teaching $\mathcal{N}$ the exact behaviour of our linear model-based fitting approach for estimating the facial expression parameters, we fine-tune our DCNN ($\mathcal{N}$) on the 4DFAB dataset [8]. The 4DFAB dataset is a large-scale database of dynamic high-resolution 3D faces (more than 1.8M 3D face) with subjects displaying both spontaneous and posed facial expressions. The ResNet [30] network structure was selected and trained after replacing the output softmax layer by a linear regression layer of $n_{e}=28$ neurons. Before starting the training, dataset frames were aligned to a template of size $224\times 224$ having the 68 points mark-up [56] projected from the mean 3D face $\mathbf{\bar{x}}$ into the image space. In total, the trained DeepExp3D is a mapping: $\mathcal{R}^{224\times 224}\rightarrow\mathcal{R}^{28}$. Note finally that 70% of the dataset was used for training and the rest were halved for testing and validation. While training, the network minimises the $\ell_{2}$ norm error between the output and the ground-truth facial expression parameters.

To classify the generated facial expression vectors $\mathbf{e}\in\mathcal{R}^{28}$ produced by the DeepExp3D network $\mathcal{N}$, Error Correcting Output Codes (ECOC) method [14] was utilised to solve this 7-class (neutral + six basic emotions) classification problem. ECOC strategy combines multiple binary learners to solve the multi-class classification problem. Our binary learner of choice is Support-Vector Machines (SVM). 10-fold cross validation with the one-versus-all [50] coding scheme were implemented to train/test the emotion classifier. The SVM hyper-parameters were optimized using the Bayesian optimization approach [58]. 68-landmarks were extracted from the images of all the employed emotion datasets in table I and used to register them to the mean face template, as done in section III-C.

To evaluate our FER method, 5 publicly available datasets for emotion recognition were used, namely: Radboud [37], KDEF [47], RAF-DB [41], CFEE [15], CK+ [46]. All five datasets have basic emotion [16] annotations (happy, sad, fearful, angry, surprised, disgusted), as well as the neutral expression. Presentation of results per dataset follows:

First of all, the Radboud dataset [37] has 67 subjects imaged from 5 different angles each at the same time. To test the performance of our network on recognising an emotion from dissimilar view angles, we run two experiments. In the first, the frontal image of each subject showing a specific emotion was kept ($67\times 7=469$ images in total), while in the second experiment one of the semi-profile faces (captured from 45/135 degrees) of each subject was selected randomly and used for 10-fold cross validation. Figure 2 reports the confusion matrices and accuracies obtained in both cases. The average MSE of expression parameters generated from semi-profile and frontal images is 0.008 over all the subjects. The comparable generated accuracies in figure 2, as well as the small MSE, demonstrate the ability of the DeepExp3D in producing view-angle independent expression estimations. As shown in table III, our proposed approach produces the highest accuracy (97.63$\%$) on the RadFD [37] dataset compared to recent state-of-the-art methods.

The KDEF dataset [47] is similar in structure to RadDF [37] where each of the 70 subjects was pictured from five different angles at the same time ($0^{\circ}$, $45^{\circ}$, $90^{\circ}$, $135^{\circ}$, $180^{\circ}$). Each subject was asked to elicit the same emotion twice, only one thereof was picked randomly. Only frontal images ($7\times 70=490$ in total) were employed in the reported results in table III. We attain the best accuracy compared to other state-of-the-art methods ($92.24\%$), revealing the power of our DeepExp3D in generating separable facial expressions according to their basic emotion label. Figure 3 demonstrates the confusion matrices generated by our emotion classifier on either frontal images (left) or semi-profile ($45^{\circ}$) images (right). Both reported confusion matrices emphasize the high separability of happy and disgusted labels from the rest of the emotions ($100\%$ and $97.1\%$, respectively), while it seems that the sad and neutral expressions tend to group closely ($78.6\%$ and $84.3\%$ for frontal, and $71.4\%$ and $78.6\%$ for semi-profile images, respectively).

The CFEE dataset [15] was collected from 230 subjects with two groups of labelled images, basic and compound. Images labelled with basic emotions (total of 1836) were passed to the DeepExp3D and then for training/testing our emotion classifier. Our obtained average accuracy per class is comparable to the state-of-the-art on this dataset by Du et al. [15], see table III.

The RAF benchmark [41] is the most challenging among all utilised FER datasets in this paper. This dataset was collected from the internet, no lab-controlled conditions. The authors of [41] sought the help of well-trained annotators for segregating the dataset into basic and compound emotion images. We use the basic emotion images, which are 13395 in total, and estimate their facial expressions using DeepExp3D. The train/test splits provided by the authors of [41] were used for training/testing our emotion classifier. Our produced average accuracy per class is comparable to the highest accuracy reported on this dataset ($82.06$ vs $83.27$).

On the Extended Cohn-Kanade (CK+) [46] dataset, our method manages to generate the highest accuracy (96.45$\%$) compared to other methods. This dataset has 327 sequences of frontal images originating from 210 subjects. Similar to [7], we keep the peak frame of each sequence and associate it with the label of this sequence.

Overall, quite similarly in all experimented benchmarks, the trained emotion classifier recognises the happy, fearful, surprised and disgusted emotions better than the rest (neutral, sad, angry). This can be mainly referred to two essential factors: 1) intensity of the related action units when deconstructing each emotion according to the Emotional Facial Action Coding System (EFACS) [20], 2) ability of employed expression basis ($U_{exp}$) in capturing the relevant action units. The trained DeepExp3D tends to capture well mouth- jaw- and cheeks-related motions (action units 6, 12, 14, 15, 16, 20, 26 [20]), e.g. lip corner puller/depressor, lower lip depressor, lip stretcher, jaw drop, etc., which are essential in characterising the happy, surprised, disgusted and fearful emotions. On the other hand, subtle details around the eyes, like inner brow raiser, brow lowerer, upper lid raiser, lid tightener, etc., which are judgmental for discerning emotions like sadness and anger, appear to be more challenging for the DeepExp3D. This can be explained by the fact that action units 6, 12, 14, 15, 16, 20, 26 are better represented in the original FaceWarehouse model utilised to annotate our collected dataset of videos presented in section III-B, as well as the 4DFAB dataset [8] used for the fine-tuning stage. Please see the supplementary materials for more results and visualisations. Additionally, the extracted landmarks might degrade the results largely if they fail to annotate the 68 targeted locations on the face with good accuracy.

In this section, we investigate the ability of our proposed framework in detecting stress conditions using only facial videos. Stress is widely conceived as a complex emotional state which can be identified by biosignals [23]. However, their recording may not always be convenient and practical for daily monitoring, thus research community pursuits stress identification only using facial cues, which constitutes a quite challenging task. Related literature is limited regarding the combination of biosignals with deep learning frameworks [32, 27], or only visual cues [24, 26]. In this work, we evaluate the performance of our method against other state-of-the-art methods in stress identification. Towards that end, we utilize the dataset (SRD’15) used in [24] which has 24 subjects (with age 47.3$\pm$9.3 years) and 288 videos in total. Each subject performed 11 experimental tasks (either neutral or stressful). The whole experiment was divided into 4 phases: 1) social exposure, 2) Emotional recall, 3) stressful images/mental task, 4) stressful videos.

The frames of each recorded video were labeled as either ’stressful’ or ’non-stressful’, according to the task under investigation. Next, our method was used to perform facial expression recognition from each frame and a 5-fold cross validation was carried out, while making sure frames coming from the same subject do not exist in both training and testing folds at the same time. The experiments were repeated 10 times and the average stress recognition accuracy of each phase is reported in table IV. Note that the first phase (social exposure) was not taken into account as it contains a task with speech which affects ‘per se’ head motility compared to a neutral non speech task as explained in [25]. For comparisons, we have also followed the same protocol and applied the method of [24] which uses head motility and the method of [27] which uses heart activity signals (IBI), with their results also shown in table IV. We observe that the proposed method achieves high accuracy and outperforms the other tested methods. This is an especially promising result for stress analysis, as our method uses only non-invasive and frame-based visual features.

Even though the final output of our proposed pipeline is the emotion class, we have also conducted detailed experiments to evaluate the intermediate steps of our framework. First of all, we qualitatively and quantitatively evaluate the accuracy of estimating the 3D expression parameters, which is the intermediate pipeline step taken as output of DeepExp3D. We test its performance on the test split of our FaceVid dataset both qualitatively and quantitatively. First of all, Figure 4 presents qualitative results of the proposed method as compared to the Ground Truth (GT) reconstructions. We show both the estimated and GT expressions on top of the mean face and the GT identity parameters. We observe that the estimations provided by our method are visually very close to the GT. Furthermore, we compare our DeepExp3D with: 1) a baseline approach following a linear shape model fitting proposed in [31], and 2) ITW, a state-of-the-art 3D reconstruction method from in-the-wild images [5]. We provide both methods with the same facial expression model (FaceWarehouse [6]) and compute the average of the Mean Squared Error (MSE) between the estimations and the ground truth over the test split. Our method achieves by far the lowest (best) MSE with 0.007, while ITW [5] and the baseline [31] obtain 0.026 and 0.098, respectively. More results and visualisations are available in the supplementary material.

Furthermore, we evaluate our approach on 3D face reconstruction from videos (Sec. III-A), which is used to construct the pseudo-ground truth annotations of our training dataset (FaceVid). This evaluation is presented in the Supplementary Material, from which it can be concluded that our proposed approach outperforms the compared state-of-the-art 3D reconstruction methods and achieves satisfactory accuracy for usage in pseudo-ground truth creation.