Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis

In recent years, multimodal sentiment analysis (MSA) has attracted significant research interest with the availability of multimodal data [21, 1]. Previous work mostly focus on designing sophisticated fusion strategies to explore inter-modal dynamics[22, 23].

Two direct ways to perform fusion are early fusion [24, 25, 26, 27] and late fusion [28, 29, 30, 15, 31]. Early fusion mainly concatenates the unimodal features at input level to conduct fusion, and late fusion tends to weighted average the unimodal decisions. Those methods outperform unimodal methods, but they can not fully explore intra-/inter-modal dynamics. With the continuous deepening of more advanced fusion methods, tensor-based fusion draws increasing attention for their high expressive power in exploring cross-modal dynamics [32, 33, 5]. Specifically, tensor fusion network (TFN) [32] and Low-rank Modality Fusion (LMF) [34] adopt outer product to learn both intra-modality and inter-modality dynamics end-to-end. Furthermore, some modality translation methods such as Multimodal Transformer [10] aims at learning a joint representation by translating source modality into the target one. Also, graph fusion is considered in Mai et al. [16], who fuses the modalities using a graph fusion network which regards each interaction as one node. More recently, BERT-based methods such as Multimodal Adaptative Gate BERT (MAG-BERT) [12] and Cross-Modal BERT (CM-BERT) [14] achieve significant boost in performance by fine-tuning BERT with cross-modal information.

However, those works are heavily dependent on sophisticated fusion strategies, which introduce more parameters and higher computational costs. Besides, compared to unimodal methods, multimodal methods with complex fusion strategies may easily suffer from overfitting due to the significantly increased number of parameters. The overfitting problem is even more severe when a limited amount of training data is available. More importantly, those methods mostly focus on exploring intra-/inter-modal dynamics to narrow the gap between modalities, but neglect to consider inter-class relationships, which are also informative to help reach more discriminatory boundaries and narrow the modality gap.

Different from them, we investigate the effectiveness of contrastive learning to design a framework HyCon, which can avoid afore-mentioned problems. Our Hycon performs a combination of various ways of contrastive learning with very direct fusion strategy. Despite the simplicity of the fusion method, our proposed model still obtains new state-of-the-art results. To sum up, in addition to considering inter-class relationships for learning more sufficient intra-/inter-modal dynamics, our contrastive learning-based model has the inherent advantage to minimize the impact of limited dataset so as to learn generalized discriminatory features. And it introduces no additional parameters for training, which is time saving and reduce the probability of overfitting.

In recent years, a resurgence of work in contrastive learning has led to major advances in many fields for its effectiveness in learning mutual information of different views of the data. Its principle is quite clear, following the idea that an anchor and a positive sample should be pulled closer in the feature space, while the anchor and negative samples should be pushed apart.

Traditional self-supervised contrastive learning requires one positive sample (e.g., an augmented version of the anchor sample) and many negative samples (e.g., randomly chosen samples from the mini-batch). Self-supervised contrastive learning methods try to learn features from different views, so as to distinguish samples from different classes [17, 18]. But they only consider one positive sample in each mini-batch and cannot leverage label information even when it is available. Unlike those methods, SupCon [19] extends the self-supervised batch contrastive approach to the fully-supervised setting, and designs supervised contrastive loss (SupCon loss). Its positive samples are drawn from samples of the same class as the anchor, rather than being data augmentations of the anchor. SupCon Loss can leverage label information, and the use of many positives and many negatives for each anchor allows us to achieve state-of-the-art performance without the need for hard-negative mining, which can be difficult to tune properly. However, we find that the learning with SupCon Loss may fall into a sub-optimal solution when negative pairs are not sufficient.

Previous works based on contrastive learning mostly only has unimodal data (i.e., images). Contrastive Bi-Modal Representation Algorithm (COBRA) [20] learns joint cross-modal embeddings based on contrastive learning paradigms, which represents the data across different modalities in a common manifold. It achieves this goal by performing supervised contrastive learning, where a positive pair is defined as the representations of data samples belonging to the same modality and class, and a negative pair is defined as the representations of two samples belonging to same or different modalities of different classes. But COBRA does not utilize many positive samples in each mini-batch, and neglects the inter-modal interactions between different samples from the same class. Thus it may miss out some cross-modal dynamics.

Inspired by the previous works, we propose a novel framework HyCon for multimodal learning. With our hybrid contrastive learning strategy, HyCon can utilize label information for supervised contrastive learning, sufficiently obtaining cross-modal dynamics while at the same time preserve inter-class relationships. We also consider how to avoid the model falling into a sub-optimal solution with a refinement term. Besides, the large amount of sample pairs allows our model to acquire better representation learning ability than existing methods. In a word, our model can simultaneously preserve intra-/inter-modal interactions and inter-class relationships by constrastive learning in a hybrid manner.

Our task is to perform multimodal sentiment analysis with multimodal data by scoring the sentiment intensity. The input to the model is an utterance [35] (i.e., a segment of a video bounded by pauses and breaths), each of which has three modalities, i.e., audio ($a$), visual ($v$), and language ($l$). The sequences of acoustic, visual, and language modalities are denoted as $\bm{u^{a}}\in\mathbb{R}^{T_{a}\times d_{a}}$, $\bm{u^{v}}\in\mathbb{R}^{T_{v}\times d_{v}}$, and $\bm{u^{l}}\in\mathbb{R}^{T_{l}\times d_{l}}$, where $T_{a}$, $T_{v}$ and $T_{l}$ represent the length of the audio, visual and language modality, respectively, and $d_{a}$, $d_{v}$ and $d_{l}$ denote the dimensionality of the audio, visual and language modality, respectively.

In this section, we describe the pipeline of our proposed method, with its diagram illustrated in Fig. 2. Previous works mostly focus on designing fusion strategies to learn cross-modal interactions. However, they only consider intra-class (intra-sample) interactions but neglect the relationships of different samples, especially samples from different classes. Besides, the elaborately designed fusion strategies may introduce high computational costs and may lead to overfitting due to limited size of dataset. To this end, innovated by contrastive learning, we propose a novel framework HyCon to address MSA, which simultaneously considers intra-/inter-modal interactions and intra-/inter-class relationships. HyCon also preserves better representation learning ability with sufficient training pairs.

As shown in Fig. 2, given an input utterance, we first obtain the unimodal representations via unimodal learning networks. We then generate positive and negative pairs of unimodal representations according to different contrastive losses. After that, in the training stage, the model is trained with our designed contrastive learning to learn cross-modal interactions and inter-class relationships, which will be discarded at inference time. Finally, the learned unimodal embeddings are fused with simple fusion method.

To sum up, HyCon consists of the following main components:

• •

  Unimodal Learning Network. To obtain the unimodal latent space of the three modalities, we leverage Transformer [9] to extract the features of the audio and visual modalities, and BERT [36] to process the language modality of the input utterance, respectively, which can be formulated as:

  $$\begin{split}\bm{X}^{m}&=\bm{F}^{m}(\bm{u}^{m};\theta^{m}),m\in\{l,a,v\}\\ \bm{x}^{m}&=\bm{X}^{m}_{T_{m}}\in\mathbb{R}^{d}\\ \end{split}$$

  (1)

  where $\bm{F}^{m}$ parametrized by $\theta^{m}$ refers to each unimodal learning network, $\bm{u}^{m}\in\mathbb{R}^{T_{m}\times d_{m}}$ is the input sequence of modality $m$, and $\bm{x}^{m}$ is the projected unimodal representation. Note that $\bm{x}^{m}$ is the feature embedding of $\bm{X}^{m}$ in the last time step, and we only use the feature embedding in the last time step to conduct fusion and learning such that our model is suitable for handling unimodal sequences of various length. The feature dimensionality $d$ of unimodal representations is the same across all modalities.

• •

  Pair Generation. We construct each mini-batch with $K$ samples (each sample consists of audio, visual, language modalities). During the training stage, the positive pairs and negative pairs are randomly drawn from the mini-batch to train the model according to different losses. Pairs that are several times the number of samples can be generated, so that the model can maximize the use of datasets for better generalization ability.

• •

  Hybrid Contrastive Learning. As the core of our proposed method, three versions of contrastive losses operated on the encoded unimodal representations are designed to perform intra-/inter-modal learning in the training stage. With the designed losses, HyCon can sufficiently learn dynamics between and within modalities, preserve inter-class relationships and minimize the modality gap. The designed ways of contrastive learning are as follows:

  1) Semi-Contrastive Learning (SCL): SCL only considers positive pairs to learn interactions between different modalities of the same sample so as to minimize the modality gap.

  2) Intra-modal Contrastive Learning (IAMCL): IAMCL is performed in a supervised manner to learn intra-modal dynamics, considering multiple positive pairs and negative pairs in a mini-batch.

  3) Inter-modal Contrastive Learning (IEMCL): IEMCL is also performed in a supervised manner to learn inter-modal dynamics, and both IAMCL and IEMCL preserve inter-class relationships for more sufficient learning.

  The designed contrastive learning will be explained in detail in Section 3.3.

• •

  Fusion and Prediction. With our designed contrastive learning strategy, HyCon can leverage simple fusion method to reach the state-of-the-art performance. In our work, different fusion strategies (including simple and direct ways like concatenation) are adopted to demonstrate the effectiveness and generalization ability of HyCon, which is shown in Section 4. The defaulted fusion method is ‘Element-wise Addition’. Generally, the fusion and prediction of our multimodal architecture can be summarized as:

  $$y_{M}=\bm{F}^{M}(\bm{x}^{l},\bm{x}^{a},\bm{x}^{v};\theta_{M}),\ell=\left|y-y_{M}\right|$$

  (2)

  where $\bm{F}^{M}$ parametrized by $\theta^{M}$ is the multimodal network, $y$ is the ground truth label, $y_{M}$ is the predicted label, and $\ell$ is mean absolute error (MAE) for prediction.

The traditional way to perform contrastive learning is self-supervised contrastive learning. A batch of randomly sampled pairs for each anchor consists of a positive pair and $M$ negative pairs. The positive sample is an augmented version of the anchor sample, while each negative sample is an augmented version of the other samples. For each anchor sample, a simple self-supervised contrastive loss can be formulated as:

where $\bm{a}$, $\bm{p}$ and $\bm{n}$ denote the representation of the anchor, positive and negative sample, respectively.

To leverage label information for better classification performance, SupCon [19] extends the traditional self-supervised contrastive learning to the fully-supervised setting. In this setting, each batch is made up of $N$ positive pairs and $M$ negative pairs. A positive sample is any augmented version of samples from the same class, while each negative one comes from different classes. The use of many positives and many negatives for each anchor allows the model to achieve state-of-the-art performance without the need of hard-negative mining. For each anchor sample, supervised contrastive loss can be formulated as:

To apply contrastive learning in multimodal data, COBRA [20] adopts similar supervised contrastive loss to learn joint cross-modal embedding. The label information is leveraged in COBRA. For each anchor, the positive sample comes from the same class and the same modality, and the negative sample comes from the same or different modalities of different classes. The equation can be formulated as:

However, COBRA only considers one positive sample for each anchor as the way in traditional self-supervised contrastive learning. Besides, it only considers positive samples within the same modality but neglect those from different modalities. Consequently, the model may miss out some cross-modal dynamics, which hinders higher performance.

Moreover, observing Eq. 3,  4 and  5, it can be seen that the learning of existing methods may fall into a sub-optimal solution if $\bm{a}^{T}\bm{n}_{j}$ is minimized but $\bm{a}^{T}\bm{p}_{j}$ is not maximized. This may happen when $\sum_{j=1}^{M}{(\bm{a}^{m})^{T}}\bm{n}_{j}$ can be easily learned to be close to 0, especially when sufficient amount of negative pairs is unavailable.

To sufficiently learn cross-modal dynamics and inter-class relationships, while at the same time fully leverages label information, we propose a novel framework HyCon to perform hybrid contrastive learnig in a supervised and unsupervised manner for MSA. In the training stage, hybrid contrastive learning is designed to train the model, which introduces three ways of contrastive learning and their corresponding contrastive losses. And a refinement term is introduced to enable a better representation learning for positive pairs and prevent the model from falling into a sub-optimal solution. We will explain them in detail as follows:

The overall contrastive loss function is a weighted sum of IAMCL, IEMCL and SCL, which can be formulated as:

where $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are hyperparameters to constrain the contributions of the three contrastive losses. ${L}_{hybrid}$ is summed over all data samples of different modalities in a mini-batch. The overall contrastive loss together withe the prediction loss (see Eq. 2) constitute the loss function for our model:

where $\bm{L}_{overall}$ is the overall loss function, and $\ell$ denotes the mean absolute error of the predicted sentiment and the true sentiment.

In this paper, two of the most commonly used public datasets, i.e, CMU-MOSEI [37] and CMU-MOSI [15] are adopted:

CMU-MOSI is a widely-used dataset for multimodal sentiment analysis, which consists of a collection of 2199 opinion video utterances downloaded from websites. Each opinion video is annotated with a sentiment intensity from on -3 to +3. Following previous works [12, 10], we utilize 1,284 utterances for training, 229 utterances for validation, and 686 utterances for testing. CMU-MOSEI is a large dataset of multimodal sentiment analysis and emotion recognition. The dataset consists of over 20k video utterances from more than 1,000 YouTube speakers, covering 250 distinct topics. All the sentences utterance are randomly chosen from various topics and monologue videos, and each utterance is annotated on two views: emotion of six different values, and sentiment in the range [-3,3]. In our work, we use the sentiment label to perform MSA. We use 16,265 utterances for training, 1,869 utterances for validation, and 4,643 utterances for testing, which is the same as the previous works [12, 10].

In our experiments, the evaluation metrics for CMU-MOSEI and CMU-MOSI datasets are the same. We applie various metrics to evaluate the performance of each model: 1) Acc7: 7-way accuracy, sentiment score classification; 2) Acc2: binary accuracy, positive or negative; 3) F1 score; 4) MAE: mean absolute error (the lower the better); and 5) Corr: the correlation between the model’s prediction and that of humans (the higher the better).

1) Early Fusion LSTM (EF-LSTM), which concatenates the input features of different modalities at word-level, and then sends the concatenated features to an LSTM layer followed by a classifier to make prediction; 2) Late Fusion LSTM (LF-LSTM) uses an LSTM network for each modality to extract unimodal features and infer decision, and then combine the unimodal decisions by voting mechanism; 3) Recurrent Attended Variation Embedding Network (RAVEN) [38], which models interactions by shifting language representations based on the features of the audio and visual modalities; 4) Memory Fusion Network (MFN) [4], which proposes delta-attention module and multi-view gated memory network to discover inter-modal interactions; 5) Multimodal Transformer (MULT) [10], which learns multimodal representation by translating source modality into target modality using cross-modal Transformer [9]; 6) Interpretable Modality Fusion (IMR) [11], which improves the interpretable ability of MULT by introducing the multimodal routing mechanism; 7) Tensor Fusion Network (TFN) [32], which applies outer product from unimodal embeddings to jointly learn unimodal, bimodal and trimodal interactions; 8) Low-rank Modality Fusion (LMF) [34], which leverages low-rank weight tensors to reduce the complexity of tensor fusion; 9) Quantum-inspired Multimodal Fusion (QMF) [39], which focuses on the interpretable ability of multimodal fusion by taking inspiration from the quantum theory; 10) Interaction Canonical Correlation Network (ICCN) [40], which fuses with language embeddings with visual and audio features respectively to get two bimodal representations. Finally, the bimodal representations are fed to a Canonical Correlation Analysis (CCA) network to generate trimodal representation and make prediction; 11) Multimodal Adaption Gate BERT (MAG-BERT) [12], which proposes an attachment called Multimodal Adaptation Gate that enables BERT and XLNet to accept multimodal data during fine-tuning. The feature extraction method of MAG-BERT is the same as that of HyCon, which ensures fair comparison. MAG-BERT is the state-of-the-art algorithm on MSA.

For each algorithm, following Gkoumas et al. [41], we first perform fifty-times random grid search on the hyper-parameters to fine-tune the model, and save the hyper-parameter setting that reaches the best performance. After that, we train each model with the best hyper-parameters setting for five times, and the final results are the mean results of the five-time running. Note that since the codes of QMF [39] and ICCN [40] are not available, we directly present the results from their papers.

For CMU-MOSEI dataset, the input dimensionality of language, audio, and visual modality is 768, 74, and 35, respectively. While for CMU-MOSI dataset, the input dimensionality of language, audio, and visual modality is 768, 74, and 47, respectively. For feature extraction, Facet¹ is used for the visual modality to extract a set of features that are composed of facial action units, facial landmarks, head pose, etc. These visual features are extracted from the utterance at the frequency of 30Hz to form a sequence of facial gestures over time. COVAREP [42] is utilized for extracting the representations of audio modality, which include 12 Mel-frequency cepstral coefficients, pitch tracking, speech polarity, glottal closure instants, spectral envelope, etc. These acoustic features are extracted from the full audio clip of each utterance at the sampling rate of 100Hz to form a sequence that represents variations in the tone of voice across the utterance.

We develop our model with the Pytorch framework on GTX1080Ti with CUDA 10.1 and torch version of 1.1.0. Our proposed model is trained using Adam [43] optimizer whose learning rate is set to 1e-5. The modality margin $\alpha$ is set to 0.8. For convenience, when generating the positive and negative pairs, we only consider the positive and negative classes of sentiment instead of 7-class fine-grained sentiment. $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are all set to 1. The feature dimensionality $d$ is set to 50, and the sequence length $T^{m}$ is 50 for all modalities.