EventFormer: AU Event Transformer for Facial Action Unit Event Detection

Given an input video sequence $\mathcal{I}=\{I_{t}\}^{T}_{t=1}$ with $T$ frames recording facial actions, where $I_{t}$ is the $t^{\rm th}$ frame in $\mathcal{I}$. The annotations of $\mathcal{I}$ are composed of a set of ground-truth AU events ${\Phi}_{\rm{g}}=\{{\phi}_{i}=(s^{\rm{g}}_{i},e^{\rm{g}}_{i},c^{\rm{g}}_{i})|0\leq s^{\rm{g}}_{i}<e^{\rm{g}}_{i}\leq T$, $c^{\rm{g}}_{i}\in\{1,2,\dots,C\}\}^{M}_{i=1}$, where $M$ is the number of ground-truth AU events in the video sequence $\mathcal{I}$, $C$ is the number of AU classes, and $s^{\rm{g}}_{i}$, $e^{\rm{g}}_{i}$, $c^{\rm{g}}_{i}$ are the start time, end time, and AU class label of AU events ${\phi}_{i}$, respectively. AU event detection aims to detect a set of events ${\Phi}_{\rm p}=\{{\varphi}_{i}=(s^{\rm p}_{i},e^{\rm p}_{i},c^{\rm p}_{i})|0\leq s^{\rm p}_{i}<e^{\rm p}_{i}\leq T$, $c^{\rm p}_{i}\in\{1,2,\dots,C\}\}^{N}_{i=1}$ which match ${\Phi}_{\rm{g}}$ precisely and exhaustively. During training, the set of ground-truth AU events ${\Phi}_{\rm{g}}$ is used as supervision for detected events ${\Phi}_{\rm p}$, and during inference, the ${\Phi}_{\rm p}$ can be simply filtered as results.

Due to AU co-occurrence relationships, AU events in different AU classes can have near-identical or exactly identical temporal boundaries. Inspired by DETR [Carion et al.(2020)Carion, Massa, Synnaeve, Usunier, Kirillov, and Zagoruyko] which views object detection as a single set prediction problem, we view AU event detection as a multiple class-specific sets prediction problem. Instead of predicting a class-agnostic set with events of all classes, we predict multiple class-specific sets, each contains events for a specific AU class.

As noted above, AU event detection aims to detect class-specific sets of AU events, ${\Phi}_{\rm p}=\{{\varphi}_{i}=(s^{\rm p}_{i},e^{\rm p}_{i},c^{\rm p}_{i})\}^{N}_{i=1}$, from $\mathcal{I}$. If we bind AU class labels to events and search for a permutation of ${\Phi}_{\rm p}$ to match ${\Phi}_{\rm{g}}$ directly, some class mismatches caused by the multi-label property of AU event detection are hard to solve. For example, events with identical temporal boundaries but different AU labels in ${\Phi}_{\rm{g}}$ can match with any permutation of the corresponding detected events in ${\Phi}_{\rm p}$ during training, which causes unstable training and makes the class labels hard to learn. To alleviate the instability issue, we split ${\Phi}_{\rm{g}}$ into $C$ disjoint class-specific sets $\{{\Phi}^{c}_{\rm{g}}\}^{C}_{c=1}$ such that ${\Phi}_{\rm{g}}=\bigcup^{C}_{c=1}{\Phi}^{c}_{\rm{g}}$, where ${\Phi}^{c}_{\rm{g}}=\{{\phi}^{c}_{i}=(s^{\rm{g}}_{i},e^{\rm{g}}_{i},c^{\rm{g}}_{i})|\forall{\phi}_{i}~{}\mathrm{s.t.}~{}c^{\rm{g}}_{i}=c\}^{N_{c}}_{i=1}$, and $N_{c}$ is the number of ground-truth events belonging to class $c$. In this way, the problem turns into predicting several class-specific sets ${\Phi}^{c}_{\rm p}$, (${\Phi}_{\rm p}=\bigcup^{C}_{c=1}{\Phi}^{c}_{\rm p}$), one for each AU class, i.e. a multiple class-specific sets prediction problem.

For the implementation of EventFormer, assuming $N_{0}$ is a number larger than any $N_{c}$, we pad ${\Phi}^{c}_{\rm{g}}$ with $\varnothing$ (no event) to make the set $\tilde{{\Phi}}^{c}_{\rm{g}}=\{\tilde{\phi}^{c}_{i}=(s^{\rm{g}}_{i},e^{\rm{g}}_{i},v^{\rm{g}}_{i})\}^{N_{0}}_{i=1}$ with a fixed number of events, where $v^{\rm{g}}_{i}\in\{0,1\}$, $v^{\rm{g}}_{i}=0$ represents $\tilde{\phi}^{c}_{i}$ is not a valid event, i.e. $\varnothing$ for padding, and $v^{\rm{g}}_{i}=1$ represents $\tilde{\phi}^{c}_{i}$ is a valid event. We denote $\tilde{{\Phi}}^{c}_{\rm p}=\{\tilde{\varphi}^{c}_{i}=(s^{\rm p}_{i},e^{\rm p}_{i},v^{\rm p}_{i})\}^{N_{0}}_{i=1}$ as the set of $N_{0}$ detected events for class $c$, called Event Set $c$. EventFormer takes a video sequence $\mathcal{I}$ and a union of $C$ Query Sets ${\Phi}_{\rm q}=\bigcup^{C}_{c=1}{\Phi}^{c}_{\rm q}$ as inputs, and outputs a union of $C$ corresponding Event Sets $\tilde{{\Phi}}_{\rm p}=\bigcup^{C}_{c=1}\tilde{{\Phi}}^{c}_{\rm p}$.

As shown in Fig. 2, our EventFormer mainly consists of three parts.

Due to the disorder of events in one set, the loss function designed for multiple class-specific sets prediction should be invariant by a permutation of the detected events with identical class labels, i.e. in one Event Set. We apply a loss based on Hungarian algorithm [Kuhn(1955)], to find a bipartite matching between ground-truth events and detected ones for each class.

A permutation of $N_{0}$ elements $\sigma\in\Omega_{N_{0}}$ is searched by finding a bipartite matching between $\tilde{{\Phi}}^{c}_{\rm{g}}$ and $\tilde{\Phi}^{c}_{\rm p}$ for each class that minimizes the total matching cost, as shown in Eq. 1:

$$\hat{\sigma}={\arg\min}_{\sigma\in\Omega_{N_{0}}}\sum\nolimits^{N_{0}}_{i=1}\mathcal{L}_{\rm match}(\tilde{\phi}^{c}_{i},\tilde{\varphi}^{c}_{\sigma(i)}),$$

(1)

where $\mathcal{L}_{\rm match}(\tilde{\phi}^{c}_{i},\tilde{\varphi}^{c}_{\sigma(i)f})$ is a pair-wise matching cost between ground-truth $\tilde{\phi}^{c}_{i}$ and a detected event $\tilde{\varphi}^{c}_{\sigma(i)}$ with matching index $\sigma(i)$. The loss function of matching is designed to minimize the distance between matched pairs and maximize the validity of matched events at the same time. We define $\mathcal{L}_{\rm match}(\tilde{\phi}^{c}_{i},\tilde{\varphi}^{c}_{\sigma(i)})$ as

$$\mathds{1}_{\{v^{\rm{g}}_{i}=1\}}\left(\lambda_{\rm{bound}}\mathcal{L}_{\rm{bound}}(\tilde{\phi}^{c}_{i},\tilde{\varphi}^{c}_{\sigma(i)})-\lambda_{\rm{valid}}\hat{p}^{c}_{\sigma(i)}[v^{\rm{g}}_{i}]\right),$$

(2)

where $\hat{p}^{c}_{\sigma(i)}\in\mathbb{R}^{2}$ denotes the probabilities of the value of $v^{p}_{\sigma(i)}$ indicating whether $\tilde{\varphi}^{c}_{\sigma(i)}$ is a valid event, i.e. the probabilities of $v^{\rm p}_{\sigma(i)}\in[0,1]$, $\hat{p}^{c}_{\sigma(i)}[v^{\rm{g}}_{i}]$ denotes the probability of $v^{\rm p}_{\sigma(i)}=v^{\rm{g}}_{i}$, and $\lambda_{\rm{bound}}$ and $\lambda_{\rm{valid}}$ are for balancing. The boundary loss $\mathcal{L}_{\rm{bound}}$ measures the similarity between a pair of matched ground-truth event and detected event. L1 loss measures the numerical difference of the regression results, and tIoU loss measures the overlapping area ratio of matched pairs. Both of them are used, considering pairs of ground-truth event and detected event may have a minor difference in terms of L1 but a huge difference in terms of tIoU. Thus, a linear combination of tIoU loss (Eq. 3) and L1 loss is used as our boundary loss $\mathcal{L}_{\rm{bound}}$, as shown in Eq. 4.

		$\displaystyle T_{\cap}=max(0,min(e_{1},e_{2})-max(s_{1},s_{2})),$		(3)
	$\displaystyle\mathcal{L}_{\rm tIoU}$	$\displaystyle((s_{1},e_{1}),(s_{2},e_{2}))=\frac{T_{\cap}}{(e_{1}-s_{1})+(e_{2}-s_{2})+T_{\cap}}.$

	$\displaystyle\mathcal{L}_{\rm{bound}}$	$\displaystyle(\tilde{\phi}^{c}_{i},\tilde{\varphi}^{c}_{\sigma(i)})=\lambda_{\rm tIoU}\mathcal{L}_{\rm tIoU}((s^{\rm{g}}_{i},e^{\rm{g}}_{i}),(s^{\rm p}_{\sigma(i)},e^{\rm p}_{\sigma(i)}))$		(4)
		$\displaystyle+\lambda_{\rm L1}(\|s^{\rm{g}}_{i}-s^{\rm p}_{\sigma(i)}\|+\|e^{\rm{g}}_{i}-e^{\rm p}_{\sigma(i)}\|),$

where $\lambda_{\rm tIoU}$ and $\lambda_{\rm L1}$ are for balancing.

After finding a bipartite matching minimizing the matching cost, the loss function can be computed. A combination of $\mathcal{L}_{\rm{bound}}$ and $\mathcal{L}_{\rm class}$ (Eq. 5) forms the event detection loss $\mathcal{L}$, as shown in Eq. 6.

$$\mathcal{L}_{\rm class}(p^{c}_{i},\hat{p}^{c}_{\sigma(i)})=-\sum\nolimits_{v\in(0,1)}p^{c}_{i}(v)\log(\hat{p}^{c}_{\sigma(i)}(v)).$$

(5)

$$\mathcal{L}=\sum\nolimits^{C}_{c=1}\sum\nolimits^{N_{0}}_{i=1}(\mathds{1}_{\{v^{\rm{g}}_{i}=1\}}\mathcal{L}_{\rm{bound}}(\tilde{\phi}^{c}_{i},\tilde{\varphi}^{c}_{\sigma(i)})+\lambda_{\rm class}\mathcal{L}_{\rm class}(p^{c}_{i},\hat{p}^{c}_{\sigma(i)})),$$

(6)

where $p^{c}_{i}\in\mathbb{R}^{2}$ is the one-hot encoding of $v^{g}_{i}$ for $\tilde{\phi}^{c}_{i}$, and $\lambda_{\rm class}$ is for balancing. It is worth mentioning that all the detected events are involved in the calculation of $\mathcal{L}_{\rm class}$, but only the matched events in Event Sets are involved in the calculation of $\mathcal{L}_{\rm{bound}}$.

In the inference stage, bipartite matching is disabled. By given a threshold $\tau$, we can simply filter out events with $\hat{p}_{i}$ lower than the threshold and preserve more valid events. For each Event Set $c$, each preserved event $\tilde{\varphi}^{c}_{i}$ is assigned with an AU class label $c$ to form one final detected event ${\varphi}_{i}=(s^{\rm p}_{i},e^{\rm p}_{i},c^{\rm p}_{i})$ with $c^{\rm p}_{i}=c$ in ${\Phi}_{\rm p}$. In this way, the set of final AU events ${\Phi}_{\rm p}$ can be easily obtained.

We classify AU event detection schemes into three categories, which use frame-level
(Frame2Event), unit-level (Unit2Event), and video-level (Video2Event) results to detect AU events respectively. To demonstrate the effectiveness of EventFormer, which follows the scheme of Video2Event, two methods following other schemes are selected for comparison. For a fair comparison, all methods including EventFormer apply the same AU feature encoder pre-trained using frame-level AU occurrence labels.

Table 2 shows sensitivity analysis of hyper-parameters, including the number of queries in Query Set $N_{0}$, embedding dimension $d_{\rm m}$ and the number of layers $L$ of encoder and decoder. As the number of queries in Query Set increases, mAP decreases while AUC increases, due to the trade-off between mAP and AR. We notice that the mAP does not decrease a lot from $N_{0}=10$ to $N_{0}=100$, and AUC increases much slower when $N_{0}>100$. Thus, we choose $N_{0}=100$ for EventFormer. As for $d_{\rm m}$, AUC reaches its peak when we set $d_{\rm m}$ to 256, while at the same time, mAP is also around its best score. As for the number of layers $L$, we notice that EventFormer achieves better performance with $L$ increasing. For the balance between computing complexity and model performance, we choose $L=6$ for EventFormer.

To show the superiority of viewing AU event detection as a multiple class-specific sets prediction problem instead of a single class-agnostic set prediction problem, We implement a class-agnostic version of EventFormer for comparison, which generates events with AU class labels directly and applies bipartite matching once between the ground-truth events and detected ones of all classes. From Fig. 3 we can see that the class-agnostic version obtains poor results on AU2, AU15, AU23, and AU24, of which the [email protected] and AR@100 are near zero. The results variance among AU classes is huge for the class-agnostic version, while the class-specific version achieves relatively balanced results. We attribute the superiority to the binding between sets and AU classes, which is essential for stabilizing training and alleviating the variance of the results among AU classes caused by data imbalance problem.