Catching Elusive Depression via Facial Micro-Expression Recognition ^††thanks: To appear in IEEE Communications Magazine 2023, under the Feature Topic “Affective Computing and Communications”.

We use image sequences extracted from videos to detect miniature facial movements. In a video clip, each frame sequence has its onset, apex and offset frames annotated as well as its observed facial action units (e.g, upper lid raiser; cf. Section II). We observe that facial actions can play an important role in emotion recognition. Hence in this study, we divided each facial image into five region-of-interests (ROIs): eyebrow, eye, middle, lip and bottom, where middle ROI consists of the cheek and nose area, and bottom ROI consists of the jaw and chin area. In order to extract these facial ROIs for micro-expression recognition, we employed facial landmark detection which allows us to locate different facial components. To this end, we first obtain different face landmark locations using Python dlib face landmark detection model based on Histogram of Oriented Gradients (HoGs), and then determine the upper and lower boundaries of each ROI.

For each frame sequence in a video clip, we want to extract 5 ROIs for each frame. Despite changes between frames, we note that facial landmark locations remain invariant across frames. As all the frames in a sequence have the same size, we can therefore exploit the facial landmarks from the onset (beginning) frame, which is used as a “registration” image, and use those landmarks as reference points to further extract ROIs in subsequent frames up to the offset (stop) frame. Thus, for each image sequence, we construct five sub-sequences using the five extracted facial ROIs. Then, we pass each sub-sequence of images to our feature extraction phase separately, described next.

In our pilot study, we used single apex frame from each image sequence to extract useful spatial features. The apex frame, as the name suggests, contains the most information in a facial event or action during a short time duration. We employed basic Local Binary Pattern (LBP) [14] which is an efficient texture feature descriptor that has been widely used in tasks such as facial expression recognition. Its computational simplicity and robustness in describing local spatial patterns and grayscale-level changes has made it popular in this field. In LBP, with predefined radius and number of neighboring pixels, we can obtain the binary pattern of each central pixel in an image by thresholding all of its neighboring pixels. Given a radius of 1, we can then obtain a binary pattern through comparison between intensity value of central pixel and each of its 8 neighboring pixels - 1 if the neighboring pixel is greater than the central pixel, otherwise 0. Then, by converting from binary to decimal, we obtain a number to represent the central pixel. Thus, by repeating the above procedure for every pixel in an image, we obtain a histogram of the image by counting the occurrence of each possible decimal value.

However, with LBP we were only able to achieve 43% accuracy in classifying all the emotions. We identified that the reason was that spatial features itself is only effective for detecting emotions that can be recognized from obvious facial expressions (e.g., eyebrow raise and eye opening is a clear indicator of surprise). In contrast, micro-expressions have much lower intensity and are much more subtle, thereby rendering LBP much less effective.

Therefore, we take into account an additional dimension that represents temporal features within each sequence of frames. To this end, we use Local Binary Pattern on Three Orthogonal Planes (LBP-TOP) proposed by [15], to extract spatial-temporal features from image sequences (videos). In LBP-TOP, we need to perform LBP operation on each of the XY, XT and YT planes, where T stands for the time axis, via which we obtain three histogram features and then concatenate them to form the final image feature map. A histogram is a type of feature descriptor representing distribution of all possible pixel values in an image. LBP-TOP brings feature description from two-dimensional to three-dimensional space and enhances image analysis that heavily depend on temporal pattern extraction.

In our study, we partition each image sequence using a 5x5 or 10x10 grid and then extract features using LBP-TOP. A 5x5 partition divides each image in a sequence into a grid of 5x5 blocks, each of size (height/5, width/5). Then, after extracting LBP-TOP features from each of the 25 blocks, we convert the features into a histogram for each block as follows. Since each pixel (except for those at the boundary of an image) in a block has 256 possible BP patterns ranging from 0 to 255, where 255 corresponds to the binary pattern 11111111 (assuming 8 neighboring pixels), we count the occurrence of each of the 256 values to construct a 256-bin histogram. As these need to be done for each of the XY, XT and YT planes, we end up with 19200 features (flattened) for each sequence (the images sequence along the T dimension was superimposed to construct each histogram, and the 25 histograms are then concatenated). The above applies similarly to the 10x10 partition, which leads to 76800 features. After that, we perform normalization in view of the large variance in feature values.

Through the feature extraction process, we obtained histogram-like features based on value occurrences. Given the low-level histogram features extracted from each image sequence, we use both linear and nonlinear machine learning algorithms to train our micro-expression recognizer (classifier). For linear models, we use Linear SVM and Logistic Regression classifiers; for non-linear models, we use Random Forest and K-Nearest-Neighbor (KNN) classifiers. We choose them for the following reasons. SVM is well-known for its ability to solve both linear and nonlinear problems by finding an optimal hyperplane to distinguish between different classes with sophisticated data transformation. Linear kernel SVM is especially useful when dealing with a large feature set, which in our case, has 19,200 and 76,800 features. Logistic regression is powerful in analyzing correlation between dependent and independent variables. In our case, we have categorical emotion classes to which a logistic regression classifier can apply; also, its probabilistic nature and fair resistance to overfitting make it a good candidate as well. Random forest is an ensemble learning model that builds upon decision trees and uses different sampling strategies to reach a final decision. It is efficient in producing classification results for large datasets and capable of dealing with imbalanced datasets. KNN as a simple non-parametric learning model uses a distance measure between points in a space to determine similarity. Although it can be computationally costly, the output can sometimes be very accurate and reliable by virtue of its natural, distance-based similarity measures.

One of the biggest challenges in facial micro-expression recognition tasks is the limitation of available databases. There are mainly two types of micro-expression databases: posed and spontaneous. For posed micro-expression collection, participants are asked to express various emotions for given tasks. On the other hand, in spontaneous micro-expression collection, participants are asked to hide their feelings and show neutral face, while in the meantime, emotion-inducing movies are played in front of participants and videos are recorded. This method is to ensure a successful induction of spontaneous micro-expressions. Clearly, spontaneous databases mirror the real scenarios more closely and would serve detection and recognition of hidden genuine emotions better. Therefore, in this paper, we use a spontaneous micro-expression database, CASME-II [6] (accessing the dataset requires a license agreement which we have signed with the authors). This dataset contains two types of micro-expressions: those when the subjects are not self-aware and those when they intend to suppress their facial movements. There are 255 micro-expression video clips in total of 26 participants, which were preprocessed, cropped, and labeled with seven estimated emotions and their corresponding sample sizes: happy(32), surprise(28), disgust(63), sad(4), fear(2), repression(27), and others(99). The emotions are estimated by using not only participants’ self-report but also facial action units as well as the emotional content of each video that was played. The video recordings were collected using a 200 fps high-speed camera with additional illumination assistance to enhance capturing the facial area of each subject.

Fig. 1 depicts our preprocessing workflow. Image pre-processing is a common practice in computer vision to benefit feature extraction, where typical techniques include denoising, graying, enhancement, registration, normalization, etc. However, in this study, we chose not to apply denoising and enhancement and similar processes as they tend to lose very fine and subtle image details which, however, are essential to FME recognition. On the other hand, in consideration of the relatively small size of the available dataset, we grouped disgust, fear, and sadness into a new category named Negative emotions, and relabeled happiness as Positive emotion; Surprise and Others remain unchanged which are the other two categories. Negative emotions are considered as our target class in the study. For each subject, we extracted every frame sequence that corresponds to an FME event, from onset to apex to offset, after which we converted all RGB images into grayscale.

In our experiment, we considered the five facial ROIs (eyebrow, eye, middle, lip and bottom) to extract features with two block division factors 5 and 10, which correspond to a flattened feature size of 19200 and 76800, respectively. To explore potentially more effective options, we also combined facial ROIs which eventually leads to nine facial areas: 1) Whole face, 2) Eyebrow ROI, 3) Eye ROI, 4) Middle ROI, consisting of nose and cheek, 5) Lip ROI, 6) Bottom ROI, consisting of jaw and chin, 7) Eyebrow+Eye ROI, 8) Eyebrow+Lip ROI, 9) Eyebrow+Eye+Lip ROI. For each combination of ROIs, we concatenated their features. After extracting the features, we fed them into each of the four classification models: Linear SVM, Random Forest, KNN and Logistic Regression. We also performed hyperparameter tuning for each model using grid search to optimize their performance, and applied stratified sampling on our dataset due to its imbalanced nature.

We use both accuracy and one-vs-all ROC AUC as our performance metrics, as well as confusion matrix to examine specific class labels. ROC curve allows us to visualize the degree of separability of a model in distinguishing between different classes, where a curve that leans more towards the top left corner indicates better performance with a higher AUC score.

Our problem is a multi-class classification problem with a large feature space. We investigate which linear or nonlinear classifiers perform well, and which facial ROIs or combination of ROIs are the most predictive of specific emotion classes.

The results are summarized in Table I where each model has two versions depending on the block division factor (5 or 10). First by looking at the “Mean accuracy” for overall performance, we see that the Eyebrow+Lip ROI has the highest mean accuracy (65.93%) across all models, and Random Forest performs the best across all facial ROIs including whole face, with an accuracy of 64.27% when the division factor is 5. We attribute this to RF’s strong ability of handling nonlinearity which our problem heavily involves.

Next, by looking at the table vertically, we see that for Whole Face, the best performers appear at the higher division factor (i.e., 10). RF and LR classifiers both achieve the highest accuracy and, as the division factor increases, the largest performance improvement (from 62.75% to 66.67% for RF and from 60.78% to 66.67% for LR). The possible reason is due to RF’s inherent feature selection ability when the feature dimension is high, and that LR can cope well with overfitting by virtue of regularization (we use Ridge) when the feature space is large. However, for other columns that investigate ROIs instead of whole face, the lower division factor generally yields better performance. This suggests that a moderate resolution of each block is preferable to produce good detection results. Horizontally across the table, it is apparent that combined ROIs perform better than individual ROIs and even Whole Face. In particular, the Eyebrow+Lip ROI is the best performer overall, with the highest accuracy achieved (70.59%) along with RF5x5.

In summary, combining ROIs from both upper and lower facial parts (e.g., eyebrow and lip) seems to be a good choice and random forest with lower division factor would fare well due to its nonlinearity handling and feature selection capabilities. Our study points out the potential viability of using just the eyebrow+lip ROI in FME recognition, which not only reduces computational cost due to the much fewer input pixels as compared to whole faces, but more importantly is also privacy-preserving. One of the biggest concerns in facial image related applications is privacy, and our ROI-based approach offers a potential solution. Specifically, the ROI extraction procedure can be performed in a private setting (e.g., at home with a mobile app or in a private clinic), and only the ROIs will be sent out for diagnosis or research purposes (e.g., developing a dataset that contains ROI only instead of full facial images, unlike the current practice). In fact, using full facial images is the main reason why the availability of FME datasets has been so limited, and our ROI-based approach could contribute to mitigating this issue.

From the ROC curve plots in Fig. 2, we can see that eyebrow ROI compared to other facial ROIs generates uniformly good results for Negative emotions across all models. We also noticed that none of the facial ROIs can individually perform well in detecting Positive emotion whereas combination of ROIs offers better classification results. This can be seen from our confusion matrix comparison results in Fig. 3. Our target class for concealed depression detection is Negative emotion, which subsumes sad, disgust and fear. As shown in Fig. 2a, all our models have exhibited effectiveness in distinguishing them from other emotion classes. More specifically, we observed that the eyebrow+lip ROI is effective on this. See Fig. 3b where 12 out 14 Negative emotion cases are correctly predicted. Interestingly, this finding also hints toward the importance of considering both upper and lower facial areas together, as the distinguishable muscle movements from these two areas appear to contribute most to recognizing both Negative and Positive emotions in a collaborative manner.

While FME analysis has yet to be adopted for clinical diagnosis of depression due to technical and ethical concerns, we believe more private or personal applications such as at-home self-diagnosis and monitoring systems can significantly facilitate timely treatment. For example, a smart-home system with a high-resolution camera can capture user’s face at high frame rate and performs FME analysis to detect and recognize user’s implicit emotions locally. Since depression generally lasts for a long term (from weeks to years), such a system can continuously monitor user’s emotions to provide reliable feedback on her depression status. Moreover, our experiment results have shown that the combined eyebrow and lip region is representative in FME analysis, and this finding can be exploited in real system implementation to further preserve user privacy.

The property of being lightweight and making near real-time inference makes our model an ideal candidate for deployment on edge devices in self-diagnosis-like applications. Our preprocessing pipeline and training are done offline. After deployment, the costliest step in the workflow is feature extraction. Our experiment was conducted on a 64GB i7-12700H laptop. To extract LBP-TOP features, we utilized numba.jit compilation to execute code in an accelerated CPU mode. To that end, we have empirically shown (cf. Table I) that our proposed facial ROI feature combinations (e.g., eyebrow+lip) instead of whole face are effective in recognizing emotions from FMEs with only slight increase in computational cost (e.g., eyebrow+lip yields cost of 2.35s vs eye with least cost of 0.56s); the corresponding feature extraction as well as other preparatory steps on combined ROIs for each image sequence are computationally inexpensive and take only a few seconds. Moreover, our model size is rather small (the best-performing RF model consists of 25 trees among which even the deepest tree is of depth 14 only), making it feasible for edge deployment.