ICE-GAN: Identity-aware and Capsule-Enhanced GAN with Graph-based Reasoning for Micro-Expression Recognition and Synthesis

Existing databases for MER constrain the development of data-driven deep learning methods because overfitting is considered a major problem. To enlarge the current dataset, GANs can generate unseen images by modeling the training data distribution, where distribution mappings are learned by the generator and the authenticity of inputs is determined by the discriminator. So far there have been tremendous extensions of GANs [20, 21, 18], and applications of GANs to MEs are also explored [29, 30]. For instance, optical flow images are generated via GAN to enlarge the database in [29], and real facial images are produced based on AU intensity in [30]. However, our method is inspired by Auxiliary Classifier GAN (ACGAN) [19] to generate unseen MEs based on image features with controllable categories, which can be of high quality and high discriminability.

Global relations between different nodes are initially modeled by conditional random fields [31]. Recently, graph convolutional networks (GCNs) [27] are proposed and adopted in [23, 32, 33], and [34], where relational information is reasoned over graphs, and contextual interactions over textual or visual contents are captured. Besides, graph-based reasoning methods are applied in the study field of MEs. [30, 35, 36] facilitate the learning of subtle facial movements from graphs that are established on AU nodes, and structural relations are captured for geometric reasoning. However, instead of directly constructing graphs over raw data inputs, we propose to explore global relations over intermediate feature channels for representative relation learning. Inspired by SENet [25] which applies attention mechanism [26] to model the interactions between input features and channel features where spatial information is pooled, we employ self-attention within feature channels to save low-level spatial features, resulting in an attentive mapping that contains dependencies between facial attributes. Moreover, different from [30] which implements MER with supplementary information such as AUs, our work can directly learn the relational information of different face regions, reducing the amount of information required.

Different from conventional CNNs that are translation invariant, the capsule network is translation equivariant, which presents a competitive learning capacity considering the relative pose and position information of object entities in the image. The reasonable layouts of different face parts do matter during MER. Ertugrul et al. [37] manage to encode face poses and AUs at different view angles with the help of a single capsule. LaLonde et al. [38] develop a deconvolutional capsule network with U-net architecture [39] which achieves a good result in object segmentation. [8] is the first work that manages to use capsule-based architecture for MER, whose model performance is better than the LBP-TOP baseline and several CNN models. In contrast, we propose our discriminator design which is enhanced by capsule network to implement multiple tasks: to check whether the input image is real or fake and to predict the ME labels.

We propose an identity-aware generator $G$ based on the encoder-decoder structure shown in Fig. 2, which can preserve the identity feature and produce outputs with preferable classes during MES, with the help of side information. The encoding procedure explores facial attributes from onset neutral inputs $X_{on}$ via a series of hierarchical convolutional layers, resulting in intermediate feature maps $f_{i:1,2,...,6}^{enc}\in\mathcal{R}^{C_{i}\times H_{i}\times W_{i}}$ at the $i$-th convolutional layer, where $C_{i}$, $H_{i}$, and $W_{i}$ denote the channel, height, and width of the feature map, respectively.

Furthermore, we decide to utilize $f_{i}^{enc}$ by processing and propagating the intermediate information from the encoder $G_{enc}$ to decoder $G_{dec}$ for more realistic image generation, as these local features contain trivial knowledge such as 2D face geometries and non-trivial knowledge that is useful for the expression generation. So, we propose to use a reasoning module based on graphs, namely graph reasoning module (GRM), to capture facial part relations to reduce artifacts.

Instead of leveraging skip connections to directly transfer the multi-scale spatial information, we design to firstly flatten the spatial dimension and directly apply self-attention on feature channels to learn a channel map. Moreover, we propose to use graph convolution to better reason the relationships between feature channels by treating the channel map as a graph, which is named as the channel graph, so that global interdependencies between intermediate facial attributes can be captured from local feature responses.

The detailed architecture is displayed on the yellow box of Fig. 2. Based on the feature map $f^{enc}_{i}\in\mathcal{R}^{C_{i}\times H_{i}\times W_{i}}$ after convolution, we can construct a spatial graph with the node number $N_{s}=H_{i}\times W_{i}$ and node feature $C_{s}=C_{i}$. Inspired by [32], supernodes with richer expressiveness are formed by a transformation function $T(\cdot)$ to generate a new feature map $\hat{f}_{i}^{enc}\in\mathcal{R}^{C_{i}\times\hat{N_{s}}}$, and empirically, $T(\cdot)$ is chosen as a convolutional operation. To learn global relations between different facial encodings, self-attention is applied to calculate a similarity mapping of features based on $\hat{f}_{i}^{enc}$, which can be formulated as:

where both $\phi(\cdot)$ and $\theta(\cdot)$ are linear mappings. We take the similarity mapping $\mathcal{M}\in\mathcal{R}^{C_{i}\times C_{i}}$ as a channel graph, which has a set of node features $\mathbf{m}=\{m_{j:1,2,..,C_{i}}\in\mathcal{R}^{1\times C_{i}}\}$. In our case, we define $\mathcal{M}$ as an undirected and fully connected graph.

To reason over the whole channel graph, we utilize a GCN [27] to learn edge features and update node features. As the initial edge feature between two nodes is unknown and undefined, we propose to set up a self-learning adjacency matrix $A\in\mathcal{R}^{C_{i}\times C_{i}}$, which can be randomly initialized following the normal distribution and updated during back-propagation. The whole process can be expressed as:

where $\sigma$ is an activation function, identity matrix $I$ is added to construct self-looping, and $W\in\mathcal{R}^{C_{i}\times C_{i}}$ and $A\in\mathcal{R}^{C_{i}\times C_{i}}$ denote the weight and adjacency matrices, which are both learnable.

We then project the updated graph $\hat{\mathcal{M}}$ back to the same feature space of $f_{i}^{enc}$ through an inverse projection function $T^{-1}(\cdot)$. Lastly, the residual connection is employed to compensate the high-level semantics relations with the low-level information. The final output $g^{dec}_{i}$ of the graph reasoning module is represented as $g^{dec}_{i}=f^{enc}_{i}\oplus T^{-1}(\hat{\mathcal{M}})$.

The last layer of $G_{enc}$ is the bottleneck layer that learns a compressed representation of the input data, termed as identity-aware embedding $e\in\mathcal{R}^{320\times 1\times 1}$. As a controlling term for expression generation, the class label $c$ is concatenated with $e$ and random noise $z$ to form a synthesis seed $s\in\mathcal{R}^{423\times 1\times 1}$, which is fed into the decoding part where the spatial dimensions of feature maps are expanded and feature channels are reduced. Thus the decoding feature maps $f^{dec}_{i:1,2,...,6}$ can be obtained with the help of deconvolution operation $h(\cdot)$ [40]. Moreover, $g^{dec}_{i}$ will be further concatenated with $f^{dec}_{i}$ so the learned relational information from latent feature space can be leveraged to improve synthetic image quality. The $i$-th upper layer feature map (for $i\in[2,6]$) can be represented as:

Unlike CNNs operating over single scalars, capsule network attends to vectors, of which lengths are used to represent the existence probabilities of each entity in a given image. We design a multi-tasking discriminator for sample authenticity checking and ME label classification enhanced by capsule network, which enables the learning of richer visual expressions and more sensitive to the geometric encoding of relative positions and poses of entities than conventional CNNs, dubbed a capsule-enhanced discriminator.

Fig. 3 presents the detailed architecture. The facial attributes are encoded via PatchGAN for faster learning, which are then fed into PrimaryCaps to encapsulate the information at a lower level. Vectors generated from PrimaryCaps are coupled and used to activate the capsules in the next layer. Two following capsules, namely AdvCaps and ExpCaps, are designed for two separate sub-tasks: (1) AdvCaps distinguishes the real expressive images $X_{apex}$ from the synthetic ones $X_{syn}$, and (2) ExpCaps predicts the corresponding micro-expression labels of input images. Furthermore, a reconstruction network [13] is desired for a performance gain by regularizing the training of ExpCaps.

Generator Objective. A two-term identity-preserving loss $L_{ip}$ is adopted to capture the identity information embedded in $X_{on}$ (via $G_{enc}$) and thus to synthesize distinctive samples $X_{syn}$ (via $G_{dec}$). One term of $L_{ip}$ is selected as the pixel-wise reconstruction loss $L_{pixel}$ to improve the generated image quality. We empirically use L1 penalty over L2 penalty to establish direct supervision on $X_{syn}$. Meanwhile, we preserve a perceptual similarity between $X_{on}$ and $X_{syn}$ during MES as in [41] by introducing a perceptual loss $L_{per}$, which is adopted as the second term in $L_{ip}$ to preserve the facial styles with regards to different subjects. With the help of a cost network which is usually implemented as a pre-trained CNN, $L_{per}$ can be easily estimated and minimized over high-level feature representations associated with $X_{on}$ and $X_{syn}$. The overall representation of $L_{ip}(G)$ can be described as:

Discriminator Objective. The training of the capsule-enhanced discriminator is optimized by the margin loss $L_{margin}$ as suggested in [13], which can enlarge the feature distance of different facial expressions. $L_{margin}$ can be obtained via:

where $T_{k}$ = 1 if expression class $k$ exists otherwise 0, while $m^{+}$ and $m^{-}$ are the upper and lower margins and $v_{k}$ is the vector output of capsules that being activated to class $k$. Moreover, the mean square error is employed as the loss function $L_{rec}$ in the reconstruction network to regularize the training procedure. Hence, the classification-related loss $L_{cls}$ for our capsule-enhanced discriminator is summed as:

Overall Objective. As we treat GAN as our baseline, the optimization process between the generator and discriminator is described as a min-max game [18]. The learning objective $L_{gan}$ can be formulated as:

where $x$ and $y$ indicate the real images and the real labels, while $z$ and $c$ denote the random noise and fake labels.

Overall, the total loss function of our proposed ICE-GAN can be summarized as:

where $\lambda_{adv}$, $\lambda_{mes}$, and $\lambda_{mer}$ are multi-tasking weight parameters of the proposed tasks including GAN optimization, identity-aware MES, and capsule-enhanced MER.

Evaluations were conducted on the MEGC2019 cross-database benchmark, which consists of three publicly available ME datasets (SMIC [14], CASME II [15], and SAMM [16]), with a total of 442 ME video sequences (164, 145, and 133 clips collected from SMIC, CASME II, and SAMM, respectively) captured from 68 subjects (16, 24, and 28 subjects from SMIC, CASME II, and SAMM, respectively). Frames in all 3 datasets are categorized into 3 classes of positive, negative, and surprise with the Leave-One-Subject-Out (LOSO) validation method. LOSO is conducted for subject-independent evaluation, i.e., the evaluation is repeated each time for all subjects accordingly, until each subject is split alone as the 1-subject testing dataset and the remaining subjects as the training dataset. In our case, we firstly conducted experiments on all 3 datasets for cross-database evaluation (CDE) based on the Unweighted F1-score (UF1) and Unweighted Average Recall (UAR), making a fair performance comparison as in [12]. We then evaluated model performance only on CASME II and SAMM for single-database evaluation (SDE) using F1-score following [30].

In this section, we reveal detailed implementations of data preprocessing and model parameter settings for our design.

Data Preprocessing. We firstly cropped out face regions by conducting facial landmark detection repeatedly to locate and refine the positions of 68 facial landmarks [42], ending up with a bounding box decided by the refined 68 landmarks with toolkit ¹¹1https://pypi.org/project/face-recognition/. We then followed [8] to find the onset and apex frames in SMIC because annotated positions of onset and apex frames are only available in CASME II and SAMM. Moreover, we chose the neighboring four images around the apex image for each subject to augment the existing data, where we assumed that they all have the same expressive intensity. Finally, these cropped face images were resized to 128$\times$128 in grayscale.

Model Parameters. The multi-tasking weight parameters in (8) are set as follows: $\lambda_{adv}$ = 0.1, $\lambda_{mes}$ = $\lambda_{mer}$ = 1. Two reweighting ratios $\alpha$ and $\beta$ are set to 0.1 and 5e-4. We set the noise dimension $N_{z}$ to 100 and class label dimension $N_{c}$ to 3 as we only have 3 classes. The discriminator includes a 70$\times$70 PatchGAN followed by $N_{prim}$ = 8 PrimaryCaps with dimension $d_{prim}$ = 16 and two sub-capsules for classification purposes ($d_{adv}$ = 256 and $d_{exp}$ = 32). $m^{+}$, $m^{-}$, and $\lambda_{k}$ in (5) are set to 0.9, 0.1, and 0.5. Adam is selected as the optimizer to train our network, with momentums $b_{1}$ and $b_{2}$ set to 0.9 and 0.999, respectively. The learning rate is initialized as 1e-3 and decays using a cosine annealing schedule. The batch size is set to 16 with 100 training epochs. The end-to-end training procedure of ICE-GAN was implemented in Pytorch with one Nvidia RTX2080Ti GPU.

Quantitative Results. We firstly evaluated our model performance with state-of-the-art methods from MEGC2019 benchmark [12] using CDE. The results of UF1 and UAR are reported in Table I. It can be observed that ICE-GAN outperforms approaches from MEGC2019 benchmark, where UF1 and UAR scores are improved by 10.9% and 12.9% compared to those of the winner [7].

The baseline (i.e., LBP-TOP) in Table I adopts hand-crafted features and receives a lower result compared to deep learning methods. CapsuleNet [8] utilizes capsule network to achieve an acceptable result, and the remaining works (i.e., [10, 11, 7]) propose their designs based on optical flows. The superior performance of our design can be attributed to the expanded size of the database and the global relations of facial attributes which are learned on channel graphs. Although ME-Recognizer [7] captures spatiotemporal information from facial movements between onset and apex images, the intra-class information is ignored for each subject.

Our model also achieves the highest scores for individual dataset evaluation. For SAMM, ICE-GAN overpasses ME-Recognizer by 13.4% in UF1 and 23.5% in UAR. The performance in CASME II has improved by 8.0% and 10.2% for UF1 and UAR respectively. However, the improvement on SMIC is not as large as the previous two datasets because there is no clear annotation about onset or apex frames, which cannot give representative information about the subject.

We then compared our method with the latest state-of-the-art works following SDE. The model performance of F1-score for 3-category classification is reported in Table II. Compared to MicroAttention [24], since we capture the long-ranged interactions between different facial regions based on graph reasoning, our method achieves a higher F1-score with 8.5% improvement on CASME II and 54.0% on SAMM. MER-GCN [35] and AU-GACN [30] are graph-based methods that reason over AU nodes, while our ICE-GAN achieves a better performance with F1-scores improved by 64.8% and 43.9% compared to AU-GACN, which indicates that our way of constructing graphs over intermediate feature channels can give more representative relational information between facial attributes than directly reason over AU graphs.

Qualitative Results. Furthermore, qualitative results for MES are shown in Fig. 4, where comparisons between synthetic and real images of four different subjects are demonstrated. The results indicate that the generated faces achieve no significant artifacts and are basically at the same level as real samples regarding the authenticity, where light conditions are preserved in the images as well.

To better examine synthetic samples, we implemented Norm2-based difference between $X_{syn}$ and $X_{on}$ and focused on AU-related face regions. Fig. 5 presents activated regions shown in red boxes for positive and surprising faces, while negative expressions have more than one kind of MEs and therefore their patterns are difficult to be visualized explicitly. Practically, facial expressions relate to many muscle movements, so multiple AUs can be activated simultaneously. For positive classes, green boxes drawn around the eye area indicate the activation of AU6, and the ones around the lip corner indicate AU12 and AU25. For surprising samples, AU1, AU2, and AU5 are activated, which are all prototypical AUs that are critical for the recognition of positive (i.e., happy) and surprise micro-expressions, according to FACS [3].

Extensive experiments were implemented to validate the component design of ICE-GAN. The following studies were examined on the full dataset including SMIC, CASME II, and SAMM, which was further randomly split into a training set with 48 subjects and a testing set with 20 subjects.

Analysis of Generator. Three models were built to verify the effectiveness of our generator in Table III. Model A is the baseline that just uses a discriminative model for MER. Model B includes a DCGAN-like generator [20], where UAR and UF1 are increased by 53.2% and 56.0% compared to model A, which validates the usefulness of GAN to expand the data size. Model C achieves UAR of 0.705 and UF1 of 0.707 based on an encoder-decoder like structure, which proves the advantage of encoding expressive representations of the input data.

Analysis of Graph Reasoning Module. To validate the excellence of the graph-based reasoning module, we compared its performance with skip connections (model D) and squeeze-and-excitation (SE) module [25] (model E) in Table III. Skip connections simply propagate low-level multi-scale spatial information from encoder to decoder side, however, there is no relation learning or reasoning on latent feature space. Thus, model D can only achieve UAR of 0.717 and UF1 of 0.724. Results shown in model E indicate that SE module helps the model modulate the channel interdependencies, and model capability is increased compared to model D with only skip connections. Moreover, by constructing graphs based on feature channels and learning global relations on local facial features, the model performance has been further improved in the final design (model F). Our model achieves the best performance with UAR of 0.761 and UF1 of 0.769, which indicates that relational information between different facial features reasoned from local responses contributes significantly when generating unseen faces.

Analysis of Synthesis Quality. Moreover, synthetic image qualities from models B, C, and F are visualized in Fig. 4. The differences between models B and C demonstrate that finer identity-related attributes can be preserved via the encoder-decoder architecture. With the help of graph-based global reasoning, more high-frequency signals can be passed smoothly through multi-scale connections (e.g. light condition) and artifacts are largely reduced.

Analysis of Discriminator Design. We then exploited the impact of the dimension $d_{exp}$ of ExpCaps on MER, within a range of [8, 16, 32, 64, 128]. As observed in Table IV, we obtained the best performance when setting $d_{exp}$ to 32, with UAR of 0.761 and UF1 of 0.769. Besides, we compared the capsule-enhanced design with its CNN-based counterparts by replacing the two-layer capsule network with a two-layer CNN, while maintaining the architecture of PatchGAN. The performance of CNN-based discriminator is reported in the first row in Table IV. To make a fair comparison of the difference in terms of model size, we increased the number of neurons in the two-layer CNN and examined the performance of the enlarged CNN-based discriminator (second row) on the MER task. As reported in Table IV, our capsule-enhanced discriminator outperforms its CNN counterparts by a large margin in terms of UAR and UF1, validating that translation equivariance introduced by capsule helps MER.