Two-Aspect Information Fusion Model For ABAW4 Multi-task Challenge

The main function of ROI Feature Extraction is to encode important information in the images. We adopt the same module in [1]. The feature extractor is an Inception V3 model, images are fed into it and generate feature maps. After that, feature maps generate spatial attention maps for the model’s regions of interest by using three convolutional layers and a sigmoid calculation. The attention maps are fused with feature maps to generate feature vectors of different regions by different encoding modules shown by red tensors. More details about ROI Feature Extraction are in [3].

The main function of Interaction Module is to extract information from different semantic spaces, then provide a more comprehensive representation of emotion descriptors for different tasks through information interaction operations. Coding through the ROI Feature Extraction module, each image is transformed into $U$ feature vectors to represent the information of the entire face. These representations are combined with positional encoding vectors and fed into two Transformer blocks to learn information from two perspectives. The outputs of these two blocks are concatenated together as an overall information representation shown by blue tensors. We assign a separate trainable query vector to each task. Each task separately performs query operation on the overall representation and integrates information from different perspectives.

The main function of Temporal Smoothing Module is to add the time domain information to the feature vector. Since the previous modules are calculated on a single image, the temporal information of the video data cannot be taken into account. Therefore, we perform a temporal smoothing operation on the encoded features. After extracting feature vectors from a piece of video data, for the feature vector $v_{t}$ at the time step $t$, we smoothed it with this function:

where $f_{t}$ is the feature that is fed to the classifier at time $t$. Unlike [1], we add the temporal smoothing operation in the training phase and assign a trainable $\mu$ to each task. It is important to emphasize that we set a trainable $f_{-1}$ for each task for the case $t=0$. To better train $\mu$, we discard videos with less than 10 frames of data. Finally, we feed the smoothed features into simple FC layers for classification or prediction.

We use BCEloss as the inference loss for the AU detection task, which formula is:

where $y_{pre}^{AU}$ denotes the AU prediction result of the model, $y_{i}^{AU}$ denotes the true label for AU of class $i$.

For the EXPR task, we use the cross-entropy loss as the inference loss, with the following equation:

where $y_{pre(i)}^{EXPR}$ denotes the EXPR prediction result of the model of class $i$, $y_{i}^{EXPR}$ denotes the true label for EXPR of class $i$.

Finally, we use the negative Concordance Correlation Coefficient (CCC) as the inference loss for the VA prediction, with the following equation:

The sum of the three loss values is used as the overall evaluation inference loss of the multitask model:

Static dataset s-Aff-Wild2 [5, 6, 7, 8, 9, 10, 11, 12, 13, 14], for multi-task learning (MTL) [4], is provided by the ABAW4 competition. It contains only a subset of frames from the Aff-Wild2 dataset. Each frame has a corresponding AU, EXPR and VA labels.

However, there are abnormal labels in this dataset, the cases of $-1$ for EXPR label, $-5.0$ for VA label, and $-1$ for both AU labels, respectively. In our experiments, to balance the number of samples between multiple tasks, we did not use the data with abnormal labels in the training and validation phase.

The aligned faces are provided by this Challenge. Each image has a resolution of $112\times 112$. We resize it to $224\times 224$ and feed it into the model. In the ROI Feature Extraction module, we try different settings and choose an implementation with U of 24 and D of 24. Transformer blocks share the same architecture, containing 4 attention head. The hidden units of the feed-forward neural network in these blocks are set to be 1024. The optimizer we used is Adam, and the total number of training epochs is 100.

We use the averaged F1 score of 12 AUs as the evaluation score for AU detection, the averaged macro F1 score as the evaluation score for EXPR prediction, and the CCC value as the evaluation score for the VA task.

In the ABAW4 challenge, the official baseline model is provided which use pre-trained VGG16 network to extract features, and 22 simple classifiers for multitask(2 linear units for VA predictions, 8 softmax activation function units for EXPR predictions and 12 sigmoid activation function units for AU predictions). The experiment results are shown in table 1, demonstrate the superiority of our model over the baseline model.

We try to add a Bidirectional Temporal Smoothing operation (BTS) to each task, but only assign a trainable initial state vector to the forward operation, i.e. the reverse operation all starts from the penultimate time step. The experiment results are shown in table 1, it is not as effective as Temporal Smoothing operation (TS).

We try to use the same structure as Sign Vehicle Space and Message Space of the SMM-EmotionNet [1] (SMM) to extract feature vectors separately before feeding them into the Temporal Smoothing and Classifier Module, but the results are not satisfactory.

We try to use only one Transformer block (Single Perspective) in the Interaction Module to extract information, and the performance degradation is significant compares to two Transformer blocks (Double Perspective). This also proves that the facial emotion descriptors information extracted by two Transformer blocks is more complete. The experiment results are shown in table 1.

To verify the effectiveness of these two Transformer blocks, we extract some samples on the validation set for inference and use the t-SNE algorithm for dimensionality reduction to visualize the outputs of these two blocks. The results are shown in Figure 2.

The results show that the sample points of the two colors are clearly demarcated, indicating that there are different types of perspective information extracted from the two Transformer blocks.

The performance of our model on each task is shown in Table 2.