Improving Multimodal Emotion Recognition by Leveraging Acoustic Adaptation and Visual Alignment

Based on the findings of the baseline study (Lian et al., 2024), we select the HuBERT-large model (Hsu et al., 2021), which has demonstrated superior performance in emotion recognition tasks, as the feature extractor. Since pre-trained transformer models (e.g., HuBERT and wav2vec2.0 (Baevski et al., 2020)) capture unique hidden representations across different layers during audio processing and that these layers may contain complementary information, we utilize the temporal pooling features ($f^{i}_{a}\in\mathbb{R}^{\mathrm{d}_{\mathrm{a}}},i\in{1,2,...,k}$) from $k$ consecutive middle layers of the HuBERT-large model for the emotion recognition task.

To adapt these features more effectively to the emotion recognition task while maintaining the generalization capability of the pre-trained model, we introduce adapters in these $k$ transformer layers for efficient parameter fine-tuning. The implementation details of the adapters are depicted in Figure 1(Stage 1). Each adapter consists of a bottleneck structure, including a dimension reduction projection layer that reduces the hidden dimension from $\mathrm{d}_{\mathrm{a}}$ to $\mathrm{\hat{d}}$, followed by a ReLU non-linear activation function, and then an up-projection layer that restores the dimension back to $\mathrm{d}_{\mathrm{a}}$. Given an input feature $x$, the output $y$ of the adapter can be represented as:

where $W_{down}\in\mathbb{R}^{\mathrm{d}_{\mathrm{a}}\times\mathrm{\hat{d}}}$, $W_{up}\in\mathbb{R}^{\mathrm{\hat{d}}\times\mathrm{d}_{\mathrm{a}}}$, $b_{down}\in\mathbb{R}^{\mathrm{\hat{d}}}$, $b_{up}\in\mathbb{R}^{\mathrm{d}_{\mathrm{a}}}$ are trainable parameters.

To account for the varying contributions of different layers to the emotion recognition task, we introduce learnable weights ($w_{i},i\in\{1,2,...,k\}$) to fuse the output features of these $k$ transformer layers, resulting in the fused feature ${f}^{fusion}_{a}={\textstyle\sum_{i=1}^{k}}w_{i}\cdot{f}^{i}_{a}$, where ${f}^{fusion}_{a}\in\mathbb{R}^{\mathrm{d}_{\mathrm{a}}}$. We use two types of loss functions to optimize the model. The first is an unsupervised masked reconstruction loss, inspired by (Hsu et al., 2021), this approach predicts the masked portions of the acoustic features using contextual information to learn more robust acoustic representations. We generate masked features ${f}^{masked}_{a}\in\mathbb{R}^{\mathrm{d}_{\mathrm{a}}}$ through a multi-layer perceptron (MLP) consisting of two fully connected layers and a ReLU layer, and measure the difference between the original and masked features using the mean squared error (MSE) loss function. The second loss function is a supervised cross-entropy (CE) loss. After passing the masked features through a fully connected layer to obtain the predicted results $x^{pred}$, we use the CE loss function to calculate the loss between the predicted results and the labels $y^{label}$. These two loss functions can be expressed as:

Thus, the overall objective function for fine-tuning the model is defined as:

Due to the semantic disparity between the visual features extracted by CLIP-large (Radford et al., 2021) and the audio features, we perform pre-training feature alignment for the visual modality prior to multimodal feature fusion, as illustrated in Figure 1(Stage 2). Inspired by (Liu et al., 2023), we first pre-train a vision MLP to align visual representations with the semantic space of pre-trained speech emotion representations. During this process, the weights of the CLIP-large model and the fine-tuned HuBERT-large model are kept frozen, and only the weights of the vision MLP are trained.

We introduce an video-audio contrastive learning method. An input video $I$ is encoded by the CLIP-large model and then average-pooled to obtain $f_{\hat{v}}\in\mathbb{R}^{\mathrm{d}_{\mathrm{v}}}$. Then, an MLP maps the visual embedding to the same dimensionality as the audio embedding, resulting in mapped visual embedding $f{v}\in\mathbb{R}^{\mathrm{d}_{\mathrm{a}}}$. This transformation can be represented as:

where $W_{1}\in\mathbb{R}^{d_{v}\times\mathrm{d}_{\mathrm{a}}}$ and $W_{2}\in\mathbb{R}^{\mathrm{d}_{\mathrm{a}}\times\mathrm{d}_{\mathrm{a}}}$, $b_{1}\in\mathbb{R}^{\mathrm{d}_{\mathrm{a}}}$ and $b_{2}\in\mathbb{R}^{\mathrm{d}_{\mathrm{a}}}$ are trainable parameters of the vision MLP. The goal of the MLP is to align the visual and audio features so they can operate within a unified feature space.

To compute the similarity between the visual and audio embeddings, we first use the fine-tuned HuBERT-large model to convert the input audio $A$ into an embedding sequence $f_{a}\in\mathbb{R}^{\mathrm{d}_{\mathrm{a}}}$. The video-audio similarity can be defined as: $s(I,A)=f_{v}^{\top}f_{a}/\left\|f_{v}\right\|\cdot\left\|f_{a}\right\|$ and the audio-video similarity as: $s(A,I)=f_{a}^{\top}f_{v}/\left\|f_{a}\right\|\cdot\left\|f_{v}\right\|$. For each video and audio pair, we calculate the normalized softmax similarity scores to measure the similarities from video to audio and from audio to video:

where $\tau$ is a learnable temperature parameter. Let $y^{\mathrm{i}2\mathrm{a}}$ and $y^{\mathrm{a}2\mathrm{i}}$ represent the ground truth one-hot encoded similarities, with a probability of 0 for mismatched video-audio pairs and 1 for matched pairs. The video-audio contrastive loss $\mathcal{L}_{\mathrm{ita}}$ is defined as the cross-entropy $\mathrm{H}$ between $p$ and $y$:

Before conducting feature fusion, we outline the extraction processes for the three modalities. Acoustic Features: we utilize the fine-tuned Chinese-HuBERT-Large model (Hsu et al., 2021) to extract the speech representation $f_{a}$ for each audio sample. Visual Features: we input the facial images extracted using the OpenFace toolkit (Baltrusaitis et al., 2018) into the CLIP-large model (Radford et al., 2021) to extract the visual features. Subsequently, we use the vision MLP introduced in Section 2.2 as the feature extractor to obtain the feature representation $f_{v}$. Lexical Features: we use the Baichuan2 (Yang et al., 2023) model to extract the feature representation $f_{l}$ from the checked transcripts of the Train&Val set, as well as the subtitle files of the unlabeled data.

After obtaining the feature vectors for the three modalities $f_{m}\in\mathbb{R}^{\mathrm{d}_{\mathrm{m}}}$, $m\in(a,v,l)$, we use an MLP composed of several fully connected layers and ReLU activation functions to map each modality’s features to the same dimension. Considering the varying importance of each modality, we stack the three modality features together and calculate the attention score $\alpha$ for each modality:

where $W_{\alpha}$ and $b_{\alpha}$ are trainable parameters. The final fused feature is given by $z=h\alpha$.

Dataset: We conduct experiments using the MER-SEMI dataset, as detailed in Table 1. The Train&Val set consists of 5,030 video clips with both discrete and dimensional emotion labels. Given the absence of a predefined training/validation split, we utilize five-fold cross-validation on the Train&Val set (Lian et al., 2024). To evaluate model generalization, the MER-SEMI track includes 1,169 unlabeled video clips in a test set, drawn from a total of 115,595 unlabeled data. Participants must predict discrete emotion labels across all unlabeled data, not just the test set. The set of discrete emotion labels includes six categories: neutral, anger, happiness, sadness, worry, and surprise. We use a weighted average F1 score as the evaluation metric, aligning with the official baseline.

Implementation Details: For acoustic features adaptation, we select the 16th to 21st layers (total $k=6$ layers) of the HuBERT-large model and incorporate adapters into these layers for fine-tuning (refer to Section 3.2.1). The feature dimension $\mathrm{\hat{d}}$ within the Adapter is set to 128. During fused feature computation, we assign an initial weight of 1.0 to the 18th layer for its optimal performance, while the other layers are initially weighted at 0.0. Pre-training is conducted on the Train&Val set with a batch size of 16 and a learning rate of 1e-4. We use the Adam optimizer with a weight decay of 0.02. For visual feature alignment, the vision MLP is trained on unlabeled data with a batch size of 1024 and a learning rate of 1e-4. For the feature fusion module, features from the three modalities are mapped to 256 dimensions through an MLP for fusion, with a learning rate set to 1e-4. The dimensions of the visual, audio, and lexical features are $\mathrm{d}_{\mathrm{v}}=768$, $\mathrm{d}_{\mathrm{a}}=1024$, and $\mathrm{d}_{\mathrm{l}}=5120$, respectively.