Facial movement synergies and Action Unit detection from distal wearable Electromyography and Computer Vision

Facial expressions often co-occur with affective experiences. Smiles are among the most ubiquitous facial expressions. They are characterized by the corner of the lips moving upwards, as a resulting action of the activity of the Zygomaticus Major muscle (ZM). Although smiles have been deemed the prototypical expression of happiness, not all people who smile are happy. The so-called Duchenne marker, or movement from the Orbicularis Oculi muscle (OO), often co-occurs with the Zygomaticus Major activity. Whilst it has been claimed that the Duchenne marker is a signal of smile spontaneity [1, 2, 3, 4], other studies have found this marker in posed smiles as well [5, 6, 7]. Others suggested that it might also signal smile intensity instead of smile authenticity [8, 6, 9].

The Facial Action Coding System (FACS) [10] is a standardized method to label facial movements. It involves identifying Action Descriptors for movements involving multiple muscles from their onset to offset. The advantage of using the FACS is that no subjective inferences about the underlying emotion are made during the facial movement identification. From these AU configurations, inferences can be made by experts in the frame of different theories of emotion. In the FACS, the lip corner pulling upwards is labeled as Action Unit 12 (AU12), and the movement around the eyes in the form of a cheek raiser is labeled as AU6. AU6 is also the AU associated with the Duchenne Marker. AUs can be measured by visual inspection using video recordings, either by a human coder [11] or by using Computer Vision (CV) algorithms [12]. Additionally, the underlying muscle activity can be measured with Electromyography (EMG) [13, 14]. The standard method is to place the EMG electrodes directly on top of the relevant muscle to increase Signal-to-Noise Ratio (SNR). More recently, several studies have proved the feasibility of measuring facial expressions with distal EMG [15, 16]. Distal EMG refers to measuring muscle activity from a body location that is distant from the relevant muscle. Distal EMG measurements are possible through volume conduction whereby the electrical activity generated by each muscle spreads to adjacent areas [13]. By measuring EMG distally, the unnatural obstruction that the electrodes pose to the production of facial expressions is reduced. Despite this advantage, distal measurements make it difficult to know the exact location of the EMG activity source. Hence, current technology has been only used to identify grouped muscle activity such as smiles or frowns. Detecting such facial expressions from EMG has its own merit, such as high temporal resolution and robustness against occlusion. However, to compare the knowledge drawn using this technology to the large body of facial expression research that uses AUs as the basis of analysis, we need to identify muscle movement activity at the AU level.

Units of movement are often grouped to form full facial expressions. In movement science, muscle synergies are defined as joint movements produced by muscle groups [17]. Analogously, we define facial movement synergies as groups of muscles, or AUs, moving together. We hypothesize that muscle synergies might not correspond 1:1 to visible observable synergies, due to the differences between muscle activity and the kinematics of the movement. Additionally, by using synergy analysis, we should be able to observe different synergies involving AU6 and AU12 in the context of posed and spontaneous smiles. This might provide more evidence in favor or against AU6 being a marker of spontaneity. If AU6 and AU12 move together in spontaneous smiles, but not in posed smiles, the Duchenne marker might be related to smile genuineness. On the other hand, if the Duchenne marker is not a signature of spontaneity, we should observe similar synergies in both posed and spontaneous smiles. We conducted this co-occurrence analysis using human-labeled AUs, CV-labeled AUs, and EMG-labeled AUs. The objective was to find common factors in how a smile is produced, and the relationship between visible movement and EMG synergies.

For the purpose of this analysis, we propose a sensing-source-synergy framework. Blind Source Separation (BSS) methods, such as Independent Component Analysis (ICA) or pre-trained OpenFace models, could be used to go from sensing to sources. Furthermore, analyses such as Non-Negative Matrix Factorization (NNMF) can be used to identify synergies from fine-grained movement units. We explore several algorithms within this space to shed light on the spatial and temporal elements that form posed and spontaneous smiles.

Moreover, it is possible to create an AU identification system that works in recording sessions where high movement or high facial occlusion are expected by combining CV- and EMG- based methods. In those cases, CV alone would struggle to continuously identify certain AUs. On the other hand, wearable distal EMG can deal with occlusion and movement, but it cannot disentangle AUs so easily. With our proposed method, CV can be used in a calibration period as a reference for automatic AU identification in EMG. Thus, we would be able to create a more robust system that enables fine-grained analysis of facial expression synergies and activation. A use case scenario would be when assessing children’s behavior, or when developing applications for Virtual Reality. It would be possible to have a short calibration session with both camera and EMG recordings. Using this multimodal information, the EMG could be tagged and used to identify AUs without requiring constant camera surveillance while the children play and run around, or when the face is covered by a headset.

The main contributions of this work are:

1. 1.

   A framework to analyze facial activity by estimating sensed signal sources and synergies.

2. 2.

   A method to identify individual muscle activity sources linked to the AUs 6 and 12 during smile production from a multimodal system. This system uses both CV and EMG during calibration, and EMG only for high-movement, high-occlusion situations. We use CV as reference method to identify different EMG components. This method is based on Independent Component Analysis (ICA) for Blind Source Separation (BSS) of the EMG readings to estimate the source muscle. Additionally, we use Non-Negative Matrix Factorization (NNMF) to identify AU synergies.

3. 3.

   An analysis of facial movement synergies using AUs as sources. We present two selection methods to analyze co-occurring activation patterns of AU6 and AU12 in the context of posed and spontaneous smiles.

We propose a framework to analyze sensed signals by estimating their sources and synergies. Since AUs are closely related to individual muscle activity, we refer to them as “sources” (Fig. 3). Sources are facial movement units caused by a certain muscle. These individual sources often move in synchronous manner to form visible facial expressions such as smiles. A group of muscles, or a group of AUs, moving together are called synergies. Different transformations are necessary to go between sensed signals, source signals, and synergies. To go between a sensed signal mix to the source signals originating a movement, we can use ICA for EMG, and OpenFace for videos. Similarly, NNMF can be used to go from movement sources to synergy groups. A special case is when the sensed signal is very close to the source signal. For example, if EMG is measured directly from the muscle originating the muscle activity, we can assume equivalence between measurement and source. In such cases, NNMF can be applied directly to the sensed signal to obtain synergies.

This data is a subset of the data generated in a previous study [40]. Here only a brief description is provided for informative purposes.

We introduced a novel framework to estimate muscle movement sources from distal EMG. Using this framework, we analyzed facial muscle synergies in an automatic manner. Two types of data analysis were performed (Fig. 3). The first type is multimodal identification, which aims to identify different sources or muscle groups from the recorded EMG; and assesses their similitude to AUs detected using CV. We rely on feature engineering using blind-source separation, cross-correlation, and dimension reduction. This is, we are using feature engineering applied on a continuous-time series to transform the data into more relevant information that can be thresholded by its SD. The second one is synergy analysis, aimed to identify the spatial and temporal structures of the facial expressions present in the data. The synergy analyses were conducted both on single modalities and at the multimodal level. Both types of analysis used the same type of EMG pre-processing.

EMG Pre-processing. The four EMG channels were first passed through a custom Hanning window with a ramp time of 0.5 s to avoid introduction of artificial frequencies by the filtering at the start and the end of the signal. Afterwards, the signals were (1) linear detrended, (2) transformed to have zero mean and one standard deviation, (3) band-pass filtered from 15 to 490 Hz, (4) rectified, and (5) low-pass filtered at 4 Hz.

CV-based AU labeling using OpenFace. The Facial Behavior Analysis Toolkit OpenFace 2.0 was used to identify several facial features including AUs. AU identification is given both as a continuous output or intensity rating; and a binary output indicating AU presence. The intensity and presence predictors have been trained separately and on slightly different datasets, which means that they are not always consistent [12]. In this work, we choose to use the continuous or the binary rating depending on the requirements of our algorithm. The binary CV AU labels are extracted and upsampled, as well as the continuous labels, from 30 Hz to 1 kHz to match the EMG sampling frequency.

Blind-source separation. ICA [41], was used to automatically estimate different muscle activity sources. The wearable used to collect the data has four channels. Thus, we set the number of decomposed components to three.

Synergy identification. NNMF [42] is a dimensional reduction method aimed to uncover synchronized muscle movements. We expect it to be able to identify source activity happening at the same time, either from CV-AUs or EMG. If AU6 and AU12 happen at different times, they should be categorized as belonging to different synergies. The used dataset contains both posed and spontaneous smiles. Thus, we hypothesize that NNMF will be able to identify whether AU6 and AU12 belong to the same synergy or not. The number of synergies found primarily depends on the number of available sources. In the case of CV, 17 AUs were identified. Hence, the degrees of freedom were a maximum of 16. In the case of EMG, the degrees of freedom is three when applying NNMF on the raw measurements, and two when applied on the estimated ICs. Fig. 3-3 shows in detail the processing followed.

Multimodal identification with component matching to CV-generated labels. A matching method was used to assess similarity between EMG components and CV-based labels (Fig. 3-2). We assume the EMG signal to contain AU6, AU12 and noise. Noise is defined as electrical interference as well as other muscular sources. First, we calculated the cross-correlation of the three ICA components; the continuous AU6, AU12 OpenFace CV-labels; and an uniformly distributed random noise distribution. Since AU12 stems from the large and strong ZM muscle, the index of the maximum correlation is chosen to correspond to AU12. The other two ICs get assigned to be AU6 and noise in order of maximum correlation value. Afterwards, a threshold method was used to determine active samples. Further smoothing is applied on the individual ICs by means of a first order Savitzky-Golay filter with length 301. An initial period of $\approx 1~{}s$ or 30 samples of the IC is used to calculate the baseline signal average and standard deviation. The whole signal then is turned into a binary vector where samples that cross the threshold of $\overline{m}+k\sigma$ with $k=2$ are set to one. The values set to one are thought to correspond to activity of the respecting AU assigned to the IC during the process of AU identification. In this process, we were careful not to use the ground truth labels (Human-coding) as input for our algorithm. The human labels were only used to assess how good the EMG threshold method is approximating visible activity tagged by the human coders.

Synergy Analysis. In some contexts, AUs might appear together more often. This might be the case for the co-occurrence of AU6 and AU12 in spontaneous smiles, should the Duchenne marker truly be a marker of smile spontaneity. In contrast, posed smiles would be characterized by less synchronized AU6 and AU12 activity. Hence, several co-occurring activation were analyzed using different modalities (Fig. 3-3-A). We propose six activation patterns: (1) AU6 only; (2) AU12 only; (3) AU12 inside AU6; (4) AU6 inside AU12; (5) AU12 before AU6; and (6) AU6 before AU12. An algorithm was designed to detect these from binary labels. These labels can be generated by human coding, CV, EMG or a combination of them. This method relies on subtracting the labels of one AU from the other, and then identifying the differences within each block for each activation pattern.

NNMF was used to extract muscle synergies from EMG, and AU synergies from the continuous CV-labels. The total variance accounted for (VAF) was used as a metric to determine the ideal number of synergies [38]. First, we used the AUs derived from CV (Fig. 3-3-B), and decomposed them into NNMF components between 1 and 16 per experimental block. Afterwards, the ideal number of synergies was determined by ensuring that the number of synergies would reconstruct the original signal with less than 15% error for all participants in both posed and spontaneous blocks. Furthermore, we compared the EMG-based and CV-based synergy detection to match them across modalities. The matching was done in the temporal domain using cross-correlation (Fig. 3-3-D), and in the spatial domain using cosine similarity to sort and match the cross-modal NNMF component weights (Fig. 3-3-E). To use cosine similarity, it is necessary to have components with equal number of weights in both modalities. Thus, we decomposed three synergies from only four smile-related CV-labeled AUs (AU 6, 7, 10 and 12). This was to match the four EMG channels. Nevertheless, we would prefer to do a cascading transformation by first determining the sources using ICA, and then applying NNMF to the ICs (Fig. 3-3-C). However, due to the limited number of EMG channels and the evidence from the multimodal identification that AU12 and AU6 are strong in the raw signal; we considered the EMG signal to be equivalent to the sources only for this analysis (Fig. 3-3-E). Given this, the spatial matching would be an alternative to the multimodal identification described in Fig. 3-2.

Delay between raw EMG and CV-based AU detection. The delay between the EMG signals and the CV-based AU detection was calculated by looking at the lag at which the maximum cross-correlation between EMG channels and AU labels appeared. A similar method was used to calculate the delay between AU6 and AU12 from the CV-based binary labels.

Agreement with human coders. Human-coded labels, CV-labels, and EMG-labels were transformed to a matching sampling rate. Then the agreement between different measurements was calculated using Cohen’s Kappa [43]. Additionally, we report accuracy, precision, and recall. The advantage of using Cohen’s Kappa is that it penalizes for the larger amount of no AU samples in the set, given that participants did not smile all the time.

We observed co-varying activation patterns of pre-processed EMG, and continuous AUs extracted from CV. There was a delay between CV AU activation and EMG activation, with EMG activation leading by 374 ms. This was expected as EMG originates skin displacement. This delay was larger than that observed from proximal EMG measurements (average of 230 ms) [44].

We showed that AU6 and AU12 can be detected from a multimodal algorithm that labels EMG signatures estimated with ICA with labels generated automatically with CV. These EMG signatures do not always correspond to those dictated by the CV algorithm during the calibration. However, they yield to a slightly higher agreement than if CV was used alone. The main reason why the accuracy of the CV labels did not constrain the accuracy of the EMG-derived labels is that CV is used only for initial identification of the EMG components that are more correlated to each AU. Afterwards, the EMG-based identification is independent of the OpenFace labels.

We proposed the framework of Sensing-Source-Synergies. We distinguish the measurements made from the sources of the movement, and we use those sources to estimate synergies. We suggested to use ICA to search for activity from different muscle sources when measuring with EMG. On the other hand, NNMF attempts to find joint activation of different muscles. Several analysis pipelines in different sensing modalities were suggested to quantify the synergies between multiple sources. The human-coded co-occurring blocks of AU6 and AU12 showed that the Duchenne marker appeared simultaneously to the lip movements about 90% of the time, independently of the experimental block. Interestingly, AU6 leading AU12 occurred in more cases than the opposite. A similar analysis on the binary CV-based labels showed a lesser percentage of simultaneous activation, followed by an even less simultaneous activation depicted by the EMG labels. This is probably because the automatic algorithms tend to detect gaps in between AU6 activation when human coders do not report so. Nevertheless, in the cases where the Duchenne marker and the lip movent co-occurred, AU6 was active before AU12 above 80% of the time in the spontaneous block; and above 60% of the time in the posed block. The agreement of AU6 and AU12 labels per modality was the highest for human-coding, followed by CV and EMG. Probably EMG had the least co-occurence agreement because of its higher temporal resolution, and, most importantly, because muscle activations lead visible activity. This suggests that indeed, muscle synergies do not perfectly match visible facial activity.

To better interpret the amount of AU synergies, NNMF was also performed on the AU labels derived from CV, and a VAF metric was used to determine the ideal number of synergies. The results suggested four synergies. The weights suggested that not all AUs contributed to these synergies, and no differences between posed and spontaneous experimental blocks was found. Another method proposed to select synergies was temporal matching with the EMG IC components. This method yielded three synergies with similar characteristics to the ones selected with the VAF criteria. Also, no significant differences were found between posed and spontaneous blocks. This entails that even though AU6 and AU12 weights are often clustered together within the selected synergies, there were no differences between posed and spontaneous smiles. According to these analyses, participants also displayed other AUs, notably AU4. This was expected, especially in the posed block where there might be a conflict between what the participants felt and the happiness they were asked to convey with a smile. It does not mean that those AUs are part of a smile, but that they were also present in the block. This suggested that antagonistic AU synergies might also occur, as to try to inhibit or mask facial expressions other than the intended one. Furthermore, we observed that OpenFace often lead to high weights in AUs that are similar to AU 6 and 12. For example, AU6 might be confused with AU7, and AU12 with AU10. Thus, when we had to further reduce the AUs to match to the EMG results, we chose these. The results showed that AU6 and 12 are often grouped together, and in one case, AU6 was grouped with AU10. Thus, whilst CV-based identification has a higher spatial resolution to identify several AUs, it still confuses several labels. Additionally, NNMF was used to identify joint muscle activity directly from the pre-processed EMG in an spatial matching algorithm. The results were similar to those found by applying NNMF on CV AUs only. This is in line with the hypothesis that NNMF considers the ZM and the OO to move as one single synergy. A closer look into the NNMF results suggested that the aforementioned muscles move in a single synergy picked up by the lower electrodes of the wearable. Thus, probably the synergy is dominated by the ZM. This is in line with results showing the significant strength of the ZM when compared to other muscles, and it might be related to muscle length [13]. To reiterate, our proposed method is using OpenFace as a rough guide for EMG labeling. Our algorithm does not need the OpenFace to be perfect, as we consider only the dynamics of the facial activations as input for the cross-correlation. Since the EMG data contains the movement sources measured distally, and ICA estimates them blindly, CV is only used for (soft-) labeling. In later stages, the algorithm is completely independent from the CV accuracy. Evidence of this is the superior performance of EMG with respect of CV with regards to the AU detection. In other words, the performance of the OpenFace algorithm did not constrain our overall detection accuracy.

We also used the results of our proposed multimodal identification of AUs using EMG, as an input to the NNMF algorithm. Although the number of synergies that can be identified using this method is only two, the results were surprising. We had hypothesized that a challenge for ICA might be to disentangle multiple AUs from the mixed EMG given the joint activation of the ZM and the OO. However, the results showed that ICA actually boosted the NNMF synergy detection by transforming the data to the source space first. Only in this case, we found a strong difference between posed and spontaneous smile AU synergies. In posed smiles AU6 and 12 seem to operate independently, whereas a joint movement of AU 6 and 12 was observed in spontaneous smiles. This is somehow in line with the Duchenne marker hypothesis. Even though the appearance of the Duchenne marker can be simulated voluntarily, the underlying muscle synergies are distinct. The success of EMG to recognize subtle differences might be that (1) with our matching algorithm EMG already contains information from the CV-based labeling; (2) the forced use of a reduced set of synergies might have made the differences more salient.

In summary, our framework uses feature engineering to estimate AUs as muscle movement sources from distal EMG, overcoming the source muscle identification limitation. It has an advantage over relying only on visual cues because muscle activation does not translate 1:1 to movement kinematics. We used the results to analyze facial muscle synergies (NNMF) in an automatic manner; compared it to CV-only annotated data; and Human-annotated data, to understand the potential and limitations of this method. We conducted the co-occurrence analysis using coder’s data, CV, and EMG. The objective was to find common factors in how a smile is produced, and the relationship between visible movement and EMG synergies.

One of the limitations of this study was the number of electrodes provided in the EMG wearable. Whilst four electrodes provide a good trade-off between wearability, smile and AU detection, they are limited to conduct muscle synergy analysis. Therefore, we opted to determine the optimal number of synergies using CV only, and a synergy-matching strategy between EMG and CV. Increasing the electrode number will enable us to explore synergies containing more facial expressions. In this case, we opted to model mainly AU6 and AU12, and to consider other AUs in the EMG as “noise”. Furthermore, synergy analysis based on NNMF requires continuous labels. The human-labeled AUs were performed frame-by-frame, but the coders only indicated presence, not intensity. Thus, we opted to do a block analysis on single and simultaneous activation. Labeling AU intensity as well would have been useful to apply our NNMF method directly on the ground-truth labels. Fortunately, OpenFace derives both continuous and binary AU labels which were useful for our method. However, the CV-only detection still could be improved. In particular, we found out that CV might have confused AU10 for AU12; and AU6 for AU7. Finally, we conducted this study on data aimed to elicit smiles. It would be interesting to assess the synergies present in other types of facial expressions.

Our Sensing-Source-Synergy approach led to good AU identification from a multimodal system, and aided to fine-grained AU synergy analysis. Our results suggest that the Duchenne marker can be displayed in both posed and spontaneous smiles, but the underlying synergy differs. However, these results should be assessed carefully. Our method is computationally inexpensive and performs better estimating AUs than OpenFace alone. The EMG already contains the mixed activity of the movement sources measured distally. ICA estimates them blindly. OpenFace’s AU dynamics are inputs for cross-correlation with EMG, leading to a CV-aided AU label approximation. The final outcome is determined by a threshold on the EMG activity of the approximated source. Accordingly, the EMG AU detection performance is not constrained by OpenFace’s detection accuracy. Nevertheless, the agreement with human coders (i.e., the ground truth) can still be improved. Future work should aim to achieve better agreement to refine the synergy analysis presented here. Another alternative would be to use our proposed algorithm only with human-coded labels, instead of the OpenFace algorithm, in a cross-validation schema that ensures generalizability of the results, given that the labels would be also the ground truth.