A Pre-trained Audio-visual Transformer for Emotion Recognition

We adapt the Multimodal Transformer (MulT) architecture [14] for the pretraining task. At the high level, the architecture consists of 4 main components: the temporal convolutions that projects features of different modalities to the same dimension, a sinusoidal positional encoding to capture temporal information, the Cross-modal Transformers that allow one modality to pass information to another, and the standard Self-Attention Transformers that process the fused information produced by the Cross-modal Transformers. An overview of the MulT architecture is available in Figure 1 (right).

At the core of MulT is the Cross-modal Attention Block (Fig 1 left), which differs from the standard Transformer encoder layer in two ways. First, the module merges audio-visual temporal information though the Multihead Cross-modal Attention layer, with the Queries being in one modality while the Keys and Values being in another. Second, each Cross-modal Attention Block learns directly from the low-level feature sequences (i.e. $X^{[0]}_{b}$ is passed to the $b\rightarrow a$ Cross-modal Attention Block regardless of the layer position) while the standard Transformer takes intermediate-level features as input. Tsai et al. [14] empirically show that adapting low-level features for the Cross-modal Attention Block is beneficial for MulT. With only 2 modalities in consideration, we have 2 types of Cross-modal Attention Blocks ($V\rightarrow A$ and $A\rightarrow V$). Eventually, we concatenate the outputs of the Cross-modal Transformers and pass them through a standard Self-attention Transformer [4]. The outputs of the Self-attention Transformer are finally converted to the original audio and visual feature dimensions for predictions using two independent fully-connected layers.

Following prior work on pretrained Transformers [2, 3, 9], we use the masked frame prediction task to train our model. Specifically, we randomly select 15% of the frames, mask them for both the audio and visual inputs, and train the model to reconstruct the masked frames. Following [2, 3] on the selected frames for masking, we mask the frames all to zero with a probability of $0.8$, replace them with randomly selected frames with a probability of $0.1$ and keep them untouched with a probability of $0.1$. Similar to [3], we use the $L1$-Loss to measure reconstruction error. The loss for each prediction is the sum of the $L1$-loss for the audio modality and the $L1$-loss for the visual modality. We adapt the strategy of masking consecutive frames from [3] to prevent the model from exploiting local smoothness. We use dynamic masking for the training set (the masked frames of each input sequence are selected independently every time the sequence is called) and static masking for the validation set (the masked frames for each input sequence are pre-computed) to make the comparison between models’ performances fair.

The Multimodal Transformer is not end-to-end, so we need to extract acoustic and visual features as inputs to the model. Since features can be more or less powerful depending on the context of their usage and the target of our study is emotion recognition, we compare different baseline features on the CREMA-D and MSP-IMPROV datasets [12, 13]. The motivation for pretraining data with MulT is to capture and model temporal dependencies so we also want the base features to be temporally independent. Thus, even though features extracted from pretrained Speech Transformers such as [3, 15, 16] are powerful, they are not suitable to be base features for MulT.

With these considerations, we select and compare the features extracted from pretrained Facenet [17], pretrained ResNet [18] and OpenFace Action Units’ intensities [19] for the visual modality. For the acoustic modality, we compare Mel-scale spectrogram, Linear-scale spectrogram and features extracted by TRILL [20]. To extract features from FaceNet and ResNet for a given video, we extract frames from the video at a constant rate and crop the face regions before feeding them into the pretrained models. Because TRILL only provides one vector representation for each input acoustic sequence, we split the input sequence into segments that matches a specified frame rate and extract the representations of the segments. To make the comparison fair, we apply all extracted features to the same model (a single-layer Gated Recurrent Unit with a hidden size of $512$ and dropout ratio of $0.2$ with fixed initialization) for emotion classification (CREMA-D dataset) and continuous emotion estimations (MSP-IMRPOV dataset). We find that extracted OpenFace and TRILL features outperform other baseline features by a considerable margin on both datasets. On the CREMA-D dataset, OpenFace shows a gain of $17\%$ in Accuracy in comparison with the ResNet representations, and TRILL shows a gain of more than $11\%$ in Accuracy from the Linear-scale spectrogram. On the MSP-IMPROV dataset, TRILL outperforms the Linear-scale spectrogram with a CCC margin of at least $0.03$ while OpenFace outperforms the second-best baseline with a CCC margin of at least $0.14$.

Following prior work on pretraining Transformers [2, 3], we implement the pretraining task with two model settings: BASE and LARGE. For both configurations, we set the number of attention heads to $12$, the number of consecutive frames for masking to $3$ ($\sim 0.6$ sec) and the length of each processed sequence is $50$ ( $\sim 10$ sec).

The hidden sizes for each of the audio and visual modality are $288$ (BASE) and $576$ (LARGE). The sizes of the feed-forward layers in each cross-modal attention block are $1152$ (BASE) and $1536$ (LARGE). The sizes of the feed-forward layers in each Self-Attention Block are $2304$ (BASE) and $3072$ (LARGE). The BASE configuration has 6 $A\rightarrow V$ cross-modal attention blocks, 6 $V\rightarrow A$ cross-modal attention blocks and 6 self-attention blocks, which sums up to 38.3M parameters. The LARGE configuration has 8 $A\rightarrow V$ cross-modal attention blocks, 8 $V\rightarrow A$ cross-modal attention blocks and 8 self-attention blocks that totals 89.2M parameters. We train both models with the Adam optimizer [22]. The learning rate is set to $5e^{-4}$, with a linear learning rate scheduler and a warmup portion of $0.1$. Both models are trained with a batch size of $64$ for $30$ epochs.

For fine-tuning, the last elements from the outputs of the pretrained MulT are passed to a residual block followed by a fully-connected layer to make final predictions. We compare the performance of the fine-tuned MulT to 4 baseline models: Early Fusion GRU (EF-GRU), Late Fusion GRU (LF-GRU), the Tensor Fusion Network (TFN) [21] and the Multimodal Transformer without the pretrained weights initialization. It is important to note that TFN only processes static inputs, i.e., each modality of a sample is represented by a vector. However, we decide to include it as a baseline model because TRILL is originally developed to represent an audio as a whole with a vector [20]. Hence, for each video, we use the vector representation extracted from TRILL for the acoustic modality along with the average of the 17 OpenFace AU intensities for the visual modality as inputs to TFN.

To make the comparisons fair for EF-GRU and LF-GRU, we make them Bidirectional and control the hidden size as well as number of layers such that the number of parameters of these models are approximately the same with the BASE configuration of MulT. For MulT without pretrained weights, we perform experiments with both the BASE and LARGE configurations and report the better performing configuration based on the validation set. We train all of the models until early stopping occurs on the validation set.

Table 1 shows the performance of different models on the CREMA-D and MSP-IMPROV datasets. Since CREMA-D’s classes are balanced, we use accuracy as our evaluation metric. Following [23, 24, 25], we report the Mean Absolute Error (MAE) and the Concordance Correlation Coefficients (CCC) to assess the quality of the regression models on the MSP-IMPROV dataset.

The fine-tuned models outperform the baseline models by a considerable margin. For emotion recognition accuracy, we see a 5% improvement for the BASE model and 7% improvement for LARGE model in comparison with the baselines. On the MSP-IMPROV dataset, the fine-tuned models also shows improvements over the baselines on both Arousal and Valence regressions. Specifically, fine-tuning the BASE model achieves $3.2\%$ and $5.1\%$ gain in CCC for Arousal and Valence regression respectively. Although there might be discrepancies between train-validation-test set split, we find our best results (accuracy of 70.22% on CREMA-D, CCC of 0.697 and 0.692 on MSP-IMPROV Arousal and Valence regression) competitive with existing benchmarks on CREMA-D [26, 27, 28] and MSP-IMPROV [29, 24].

Since the ultimate motivation of transfer learning is to reduce the requirements on labeled data, we are interested in exploring the capability of the pretrained MulT in a limited resource setting. Figure 2 shows the performance of the models when only $N\%$ of the original training set are used for training. We can see that the performance drop curves for the pretrained models are less steep in comparison with training MulT from scratch and TFN. With only $10\%$ of the original training set (less than $500$ training samples on both datasets), finetuning the pretrained models outperforms training from scratch by at least $10\%$ for emotion recognition, and more than $20\%$ and $10\%$ CCC improvements for Arousal and Valence regression respectively. This further suggests the robustness of the pretrained weights in preventing overfitting with limited data. We can also note that TFN tends to perform better than training MulT from scratch with small training sets, which is expected because light models tend to be less susceptible to overfitting than complex ones with limited data.