Boosting Continuous Emotion Recognition with Self-Pretraining using Masked Autoencoders, Temporal Convolutional Networks, and Transformers

Inspired by Netease, we conduct pre-training of our MAE on a facial image dataset. To this end, we also curate a large-scale dataset of facial expressions to learn facial features, consisting of AffectNet, RAF-DB, FER2013, and FER+. Subsequently, the MAE model is pre-trained on this dataset in a self-supervised manner. Specifically, our MAE consists of a ViT-Base encoder and a ViT decoder. The pre-training process of MAE follows a masked-reconstruction method, where images are first divided into a series of patches (16x16), with 75% of these patches randomly masked. These masked images are then fed into the MAE encoder, while the MAE decoder is tasked with reconstructing the complete image. The loss function for MAE pre-training is pixel-level L2 loss, aiming to minimize the difference between the reconstructed image and the target image. Once self-supervised learning is completed, the MAE decoder is removed and replaced with fully connected layers connected to the MAE encoder. Subsequently, Expr labels are fine-tuned to obtain a feature extractor more aligned with the distribution of aff-wild2 data.

Videos are first split into segments with a window size $w$ and stride $s$. Given the segment window $w$ and stride $s$, a video with $n$ frames would be split into $[n/s]+1$ segments, where the $i$-th segment contains frames$\left\{F_{(i-1)*s+1},\ldots,F_{(i-1)*s+w}\right\}$.

In other words, videos are cut into some overlapping chunks, each with a fixed number of frames. The purpose of doing this is to break down the video into smaller parts that are easier to process and analyze. Each chunk has some degree of overlap with the previous and next ones so that no information in the video is missed.

We denote visual features as $f_{i}$ corresponding to the $i$-th segment extracted by pre-trained and fine-tuned ViT-Base encoder.

Visual feature is fed into a dedicated Temporal Convolutional Network (TCN) for temporal encoding, which can be formulated as follows:

$$g_{i}=\text{ TCN }\left(f_{i}\right)$$

This means that we use a special type of neural network that can capture the temporal patterns and dependencies of the features over time. The TCN takes the input feature vector and applies a series of convolutional layers with different kernel sizes and dilation rates to produce an output feature vector. The output feature vector has the same length as the input feature vector but contains more information about the temporal context. For example, the TCN can learn how the image changes over time in each segment of the video.

We utilize a transformer encoder to model the temporal information in the video segment as well, which can be formulated as follows:

$$h_{i}=\text{ TransformerEncoder }\left(g_{i}\right).$$

The Transformer encoder only models the context within a single segment, thereby ignoring the dependencies between frames across segments. To account for the context of different frames, overlapping between consecutive segments can be employed, thus enabling the capture of the dependencies between frames across segments, which means $s\leq w$.

We use another type of neural network that can learn the relationships and interactions among the features within each segment. The transformer encoder takes the input feature vector and applies a series of self-attention layers and feed-forward layers to produce an output feature vector. The output feature vector has more semantic meaning and representation power than the input feature vector. For example, the transformer encoder can learn how different parts of the image relate to each other in each segment of the video. However, the transformer encoder does not consider how different segments of the video are connected or influenced by each other. To solve this problem, we can make some segments overlap with each other so that some frames are shared by two or more segments. This way, we can capture some information about how different segments affect each other. The degree of overlap is controlled by two parameters: $s$ is the length of a segment and $w$ is the sliding window size. If $s$ is smaller than or equal to $w$, then there will be some overlap between consecutive segments.

VA challenge: We use the Concordance Correlation Coefficient (CCC) between the predictions and the ground truth labels as the measure, which is defined as in Eq 1. It measures the correlation between two sequences $x$ and $y$ and ranges between -1 and 1, where -1 means perfect anti-correlation, 0 means no correlation, and 1 means perfect correlation. The loss is calculated as Eq 2.

$$\begin{split}CCC(x,y)&=\frac{2*\operatorname{cov}(x,y)}{\sigma_{x}^{2}+\sigma_{y}^{2}+\left(\mu_{x}-\mu_{y}\right)^{2}}\\ \text{ where }\operatorname{cov}(x,y)&=\sum\left(x-\mu_{x}\right)*\left(y-\mu_{y}\right)\end{split}$$

(1)

$$\mathcal{L}_{\text{VA }}=1-CCC$$

(2)

Expr challenge: We use the cross-entropy loss as the loss function, which is defined as in Eq 3.

$$\mathcal{L}_{\text{Expr }}=-\frac{1}{N}\sum_{i}\sum_{c=1}^{M}y_{ic}\log(p_{ic})$$

(3)

where $y_{ic}$ is a binary indicator (0 or 1) if class $c$ is the correct classification for observation $i$. $p_{ic}$ is the predicted probability of observation $i$ being in class $c$, $M$ is the number of classes. The multiclass cross entropy loss function measures how well a model predicts the true probabilities of each class for a given observation. It penalizes wrong predictions by taking the logarithm of the predicted probabilities. The lower the loss, the better the model.

AU challenge: We employ BCEWithLogitsLoss as the loss function, which integrates a sigmoid layer and binary cross-entropy, which is defined as in Eq 4.

$$\mathcal{L}_{\text{AU }}=-\frac{1}{N}\sum_{i}[y_{i}\cdot log(\sigma(x_{i}))+(1-y_{i})\cdot log(1-\sigma(x_{i}))]$$

(4)

where $N$ is the number of samples, $y_{i}$ is the target label for sample $i$, $x_{i}$ is the input logits for sample $i$, $\sigma$ is the sigmoid function The advantage of using BCEWithLogitsLoss over BCELoss with sigmoid is that it can avoid numerical instability and improve performance.

All models were trained on two Nvidia GeForce GTX 3090 GPUs with each having 24GB of memory.

Table 1 displays the experimental results of our proposed method on the validation set of the VA, Expr, and AU Challenge, where the Concordance Correlation Coefficient (CCC) is utilized as the evaluation metric for both valence and arousal prediction, and F1-score is used to evaluate the result of Expr and AU challenge. As demonstrated in the table, our proposed method outperforms the baseline significantly. These results show that our proposed approach using TCN and a Transformer-based model effectively integrates visual and audio information for improved accuracy in recognizing emotions on this dataset.