Self-supervised Facial Action Unit Detection with Region and Relation Learning

Motivated by BYOL [17], we use two neural networks: the online network which is defined by a set of parameters ${\theta}$ and the target network parameterized by ${\xi}$. The target network provides a regression target to train the online network. The parameters ${\xi}$ of the target network are updated by the online network’s parameter ${\theta}$ according to Eq. 1, in which ${m}$ is the exponential moving average (EMA) parameter with decay rate.

$$\xi=m*\xi+(1-m)*\theta$$

(1)

To model long-distance dependencies, we use Swin Transformer as our backbone. Swin Transformer was selected due to its competitive results in both supervised and self-supervised works, with hierarchical and shifted windowing mechanisms that can help extract more precise facial features and save computational effort.

Given an input image ${x}$, we first generates two augmented images $x_{1}$ and $x_{2}$. These augmented images are fed into $f(\cdot)$, which are respectively named by $f_{\theta}$ and $f_{\xi}$. Within the online network, we then perform the projection ${z}_{1}^{g}=g_{\theta}(o_{g})$ from the pooled representation $o_{g}$, followed by the prediction $q_{\theta}({z}_{1}^{g})$. At the same time, the target network outputs the target projection from $t_{g}$ such that ${z}_{2}^{g}=g_{\theta}(t_{g})$. Our global loss is then defined as the cosine similarity between the prediction and the target projection as shown in Eq. 2.

$$\mathcal{L}_{\text{glo}}\triangleq-\frac{\left\langle\mathbf{z}_{2}^{g},q_{\theta}(\mathbf{z}_{1}^{g})\right\rangle}{\|\mathbf{z}_{2}^{g}\|_{2}\cdot\left\|q_{\theta}(\mathbf{z}_{1}^{g})\right\|_{2}}$$

(2)

As noted in self-supervised learning works, comparing random crops of an image plays a central role in capturing information in terms of relations between parts of a scene or an object [9]. However, recent work on self-supervised AU detection has not taken this into account. Moreover, the multi-crop strategy in conventional self-supervised object detection could undermine facial AU information. To obtain rich facial information, the primary issue is how to learn AU local features without destroying AU information. Unlike previous methods that random crop patches, RRL uses the attention map to make the model focus more on AU-specific regions, and the local feature learning module is used for AU feature refinement.

Specifically, We first detect 68 facial landmarks to initialize the attention maps of the AUs for a given input image. Landmarks specific to each action unit are defined similarly to EAC-Net [18]. We fit ellipses to landmarks as the initial regions of interest for each AU and smooth the image (Gaussian with $\sigma$ = 3), thus obtaining $K$ AU attention maps $i_{k}$ of 14x14 size, where ${k}$ = {1,…, ${K}$}. Then, We fuse the features learned by Swin Transformer with the attention maps and input them into the local feature learning network together with the global feature. We use [$o_{1}$,$o_{2}$,…,$o_{k}$] for local vectors and $o_{g}$ for global vectors, where the letter $o$ represents the output of the online network and $t$ represents the output of the target network.

Finally, we minimize the cosine distance of the global and local features corresponding to the two networks as shown in Eq. 3.

$\mathcal{L}_{\text{loc}}\triangleq-\frac{1}{2K}\sum_{k=1}^{K}\left[\frac{\left\langle q_{\theta}({z}_{1}^{k}),g_{\xi}(\mathbf{t_{g}})\right\rangle}{\|q_{\theta}({z}_{1}^{k})\|_{2}\cdot\left\|g_{\xi}(\mathbf{t_{g}})\right\|_{2}}+\frac{\left\langle g_{\xi}(\mathbf{t_{k}}),q_{\theta}(\mathbf{z}_{1}^{g})\right\rangle}{\|g_{\xi}(\mathbf{t_{k}})\|_{2}\cdot\left\|q_{\theta}(\mathbf{z}_{1}^{g})\right\|_{2}}\right]$

(3)

Similar to [19] using Optimal Transport (OT) to learn the discrimination between instances, an improved OT method is proposed here to learn the correlations of AUs.

The AU outputs from the online network [$o_{1}$,$o_{2}$,…,$o_{k}$] are considered to be the $k$ goods of the supplier, while vector [$t_{1}$,$t_{2}$,…,$t_{k}$] from the target network are considered to be the demands of the demander. In addition, we define the unit transportation cost between the demander and the supplier as the value of the AU relationship matrix. Taking [20] as the reference, we count the correlation between AU labels in the training set and get the AU relationship matrix $M_{ij}$ of AU, as shown in Fig.2. The larger the matrix value, the greater the correlation between the two AUs. Therefore, the unit transportation cost from the supplier node to the demander node is defined as Eq. 4.

$$\begin{array}[]{ll}{C}_{ij}=1-{M}_{ij}\end{array}$$

(4)

The correlation degree of each AU pair can be expressed as the optimal matching cost between two sets of vectors. The more relevant the vector, the lower the transport cost.

Following [19], the marginal weights $a_{i}$ and $b_{j}$ are expressed by the product of global vectors and local vectors. In that case, the importance of the AU region depends on its contribution to the whole face as defined in Eq. 5, where the function max(·) ensures the weights are always non-negative.

$\displaystyle a_{i}=\left\{\mathbf{o}_{i}^{T}\cdot\mathbf{t}_{g},0\right\},\quad b_{j}=\left\{\mathbf{t}_{i}^{T}\cdot\mathbf{o}_{g},0\right\}$

(5)

We use the cost matrix $C$ to calculate the optimal transmission scheme ${\pi}^{*}$, and solve the OT problem through the Sinkhorn-Knopp iteration. Then we can define the correlation loss function to learn the distinguishing characteristics as shown in Eq. 6. The loss would be minimized only if the representations of each AU are similar to the associated AU.

$$\mathcal{L}_{\text{corr}}(\mathbf{O},\mathbf{T})\triangleq-\sum_{i=1}^{K}\sum_{j=1}^{K}\frac{\mathbf{o}_{i}^{T}\mathbf{t}_{j}}{\left\|\mathbf{o}_{i}\right\|\left\|\mathbf{t}_{j}\right\|}{\pi}^{*}$$

(6)

After introducing each loss term, the overall loss is defined as following:

$$\mathcal{L}_{\text{all}}={\alpha_{1}}{L}_{\text{glo}}+{\alpha_{2}}{L}_{\text{loc}}+{\alpha_{3}}{L}_{\text{corr}}$$

(7)

where $\alpha_{1}$, $\alpha_{2}$, $\alpha_{3}$ are trade-off parameters.

Training. We conducted the self-supervised pre-training on the EmotioNet [21] dataset. The EmotioNet is collected in the wild, including a total of 950,000 images. All images are resized to 224 × 224 as the input of the network. Facial landmarks are detected with Dlib. In addition to random horizontal flipping, random grey scaling, and random Gaussian blur, the data augmentation also has random color distortions. Where, the random color distortion with different brightness, saturation, contrast, and hue, augmentation of different values is applied to two views. We employ an AdamW optimizer and the learning rate is $lr=0.05*batchsize/256$, using a cosine decay learning rate scheduler. The weight decay also follows a cosine decay from 0.04 to 0.4. The trade-off parameters $\alpha_{1}$, $\alpha_{2}$, and $\alpha_{3}$ are empirically set to 0.4, 0.6, and 1.0 respectively. The exponential moving average parameter $m$ is initialized as 0.98 and is increased to 1.0 during training.

We compared RRL with several self-supervised methods and supervised AU detection methods. Table 1 and Table 2 report the F1 score on DISFA and BP4D.

Comparison with self-supervised methods We compare our method with self-supervised methods: FAb-Net [12], TCAE [14], and TC-Net [15]. FAb-Net and TCAE followed the settings in TC-Net. Those models have all been pre-trained on the combined VoxCeleb dataset [24], in contrast to the static and in-the-wild EmotioNet dataset we employ in this paper. These two datasets consist of videos of interviews containing around 7,000 subjects. TC-Net_k=1 is the result of TC-Net when the time interval is 1. As shown in Table 1, the F1 score of our method outperforms all self-supervised methods. Even though the DISFA dataset has significant problems with data imbalance, the majority of our AU results are encouraging. The performance demonstrates the benefit of the relational learning module, which offers comprehensive AU correlation knowledge. Evaluations on BP4D are shown in Table 2. Without any temporal information, we still got identical results compared with recent self-supervised methods. In particular, AU2 (outer brow raiser), which is difficult to capture, has achieved remarkable results, suggesting that the local feature learning module can extract subtle AU changes effectively. As we can see from the results on BP4D, our method performs slightly worse than TCAE and TC-Net. This is because both of these methods were trained on video frames, and in addition, TC-Net results were obtained using an ensemble model that combined several independently learned encoders.

Comparison with supervised methods We also compare our method with state-of-the-art supervised methods, including AlexNet [25], DRML [2], and JAA-Net [4]. On top of the encoder, we simply train a straightforward linear classifier using frozen weights. Experimental results shown in Table 1 and Table 2 demonstrate that our RRL is comparable to fully supervised models. It outperforms AlexNet and DRML in BP4D and outperforms DRML in DISFA datasets. It lags behind JAA-Net, which adopts facial landmarks to jointly AU detection and face alignment. Besides, all supervised results are directly collected from the original work.

In this section, we conduct ablation experiments on BP4D dataset to explore the effect of each component in our proposed model and find the best structural configurations of the framework.

The Effectiveness of different components: As shown in Table 3, Global is a baseline method that shows the results of comparative learning of only two augmented images. The F1-score rises by $0.9\%$ after the implementation of local learning, indicating the significance of local features in AU detection. By enhancing the AU relation learning via OT, the model improves the baseline by $1.5\%$. The performance demonstrates the necessity of taking into account the unique characteristics of AUs while developing self-supervised tasks. Moreover, the average F1-score is further improved by $2.2\%$ compared to the baseline when we combine those two components. Overall, both local learning and relation learning components are essential for improving the performance of the RRL network.

The Effectiveness of Swin Transformer and Augmentation: To investigate the effect of the model backbone on the performance of AU detection, we replace the Swin Transformer backbone with VGG16 and report the results in Table 4. We can find that the use of the Swin Transformer significantly improves the detection performance because the hierarchical architecture in Swin-T extracts rich face representations. When the image augmentation is discarded, we observe that the results decline by $1.1\%$, indicating that data augmentation is crucial to self-supervised learning in natural images.