Cooperative Sentiment Agents for Multimodal Sentiment Analysis

Multimodal sentiment analysis (MSA) aims to infer individuals’ sentiment states through the cues they inadvertently emitted. These cues encompass behavioral aspects (e.g., facial expressions) and physiological indicators (e.g., electroencephalogram). With the recent surge in the popularity of smart devices, a multitude of users create and upload videos to platforms like YouTube to share their immediate moods. This exponential growth in video data has laid a robust foundation for MSA, predominantly focusing on behavioral cues. An intrinsic difficulty for MSA is to fuse heterogeneous data collected by disparate sensors for a more reliable prediction. Based on the data integration stage, existing MSA works can be categorized into feature-level fusion, decision-level fusion, and hybrid fusion.

Feature-level fusion consolidates several independent representations into a joint representation and subsequently makes decisions [1]. Decision-level fusion independently processes the features and classifiers of each modality [16]. It combines the decision vectors from all classifiers to produce the ultimate sentiment prediction. Hybrid fusion represents a combination of both feature-level and decision-level fusion techniques [9, 15]. Among the above paradigms, decision-level fusion is the most straightforward due to the consistent formats of decision vectors. However, achieving precise predictions through decision-level fusion remains challenging. First, it is hard to determine the reliabilities of modalities when fusing their decision vectors. Secondly, the independent prediction process for each modality fails to capture the relevance between multimodal signals, such as shared and complementary properties.

Unlike decision-level fusion, feature-level fusion makes the most of interacted properties across multimodal signals and combines multiple representations into a powerful joint one. It has become the most popular modality-fusion mode in the MSA task. Tensor-based, graph-based, transferred-based, and attention-based techniques are popular for feature-level fusion. Tensor-based works employ common tensor operations to integrate multimodal representations, like concatenation, outer product, etc. TFN (Tensor Fusion Network) uses a three-fold Cartesian product to establish trimodal dynamic explicitly [1]. LMF (Low-rank Multimodal Fusion) analyzes that the TFN, performing outer product on high-dimension embedding, has resulted in the computational burden. To optimize TFN, it exploits the parallel decomposition of low-rank weight tensor and input tensor to compute tensor-based fusion [2]. Analogously, MRRF uses Tuckers’ tensor decomposition to disentangle unimodal representations and remove redundant parts to reduce computation. It gets joint multimodal representations by performing the outer product on disentangled modal embeddings [3]. Some works also construct graphs to learn the joint embedding, where nodes and edges represent modalities and corresponding relationships [17]. HHMPN successively models the feature-to-label, label-to-label, and modality-to-label dependencies via passing graph messages [4]. Transfer-based technology is another popular way of learning the joint representation by transferring one modal representation into another with the encoder-decoder structure. The immediate features output by the encoder are regarded as the joint representation of the two modalities [10]. The joint representation for all modalities will be obtained by transferring signals in succession. Some works avoid the sequential transfer among modalities. They specify the dominant modality in advance and then transfer other modalities to the specified one to strengthen it. The attention mechanism is widely used for feature-level fusion since it inspires models to focus on some vital features when learning the joint representation. MFN proposes the Delta-memory Attention Network (DMAN) to learn fusion weights for disparate multimodal elements through time. It also designs the Multi-view Gated Memory module to filter noisy features [5]. MARN applies the attention mechanism to the hidden state of LSTM and aggregates multimodal representations [6]. MulT applies the intra-modal transformer and cross-modal transformers to alleviate the issue of (1) unaligned multimodal data; (2) long-time dependencies between elements across modalities [7]. GME-LSTM(A) [8] is devoted to addressing unimodal temporal dynamics and multimodal importance issues. It applies temporal attention to the output of the LSTM module to attend to the vital time steps and modal attention to emphasize the significance of each modality.

Although the above feature-level techniques have successfully merged multiple signals into a joint representation in form, they care more about modality-fusion strategies yet lack deliberation about specific features suitable for fusion. Recently, there has been a growing awareness that diverse inter-modal properties may significantly contribute to the joint multimodal representation. Several studies have divided raw signals into modality-shared and modality-unique representations prior to the fusion process [11, 12, 13, 14]. However, these methods still struggle with the challenge of capturing appropriate features from shared and unique components to accurately predict sentiment states. A novel technique that can overcome the limitations of existing fusion modes and adaptively capture significant properties from unimodal signals for the joint representation is still needed. In this paper, we propose to collaboratively deal with multiple signals to learn the joint representation through cooperative sentiment agents.

Except for the modality-fusion technique, the quality of unimodal representations also significantly impacts the MSA task. Firstly, multimodal data are collected by disparate sensors and represented using distinct digital signal formats. Video, audio, and text data, for instance, are recorded via frames, waveforms of amplitude values over time, and sets of characters, respectively. Meanwhile, each modality carries different levels of modal and sentiment properties. It is essential for MRL to disentangle sentiment features from heterogenous data and represent them in a unified format. Secondly, features from adjacent frames often exhibit severe overlap and minor value fluctuations, making them weak at reflecting temporal dynamics. Moreover, prevalent approaches typically aggregate multimodal representations initially and subsequently extract temporal features, inadvertently neglecting the exploration of unimodal sentiment variations over time. The heterogeneous multimodal data and implicit temporal dynamics have increased the difficulty for sentiment agents in learning significant cross-modal features. To alleviate the above issues, we propose the Modality-Sentiment Disentanglement (MSD) and Deep Phase Space Recognition (DPSR) modules when establishing sentiment agents.

Reinforcement Learning (RL) [21] is about an agent spontaneously interacting with the environment, taking actions according to policies, and rewarding them. The environmental state $s$, agent’s policy $\pi$, state transition $p$, and reward function $r$ are essential components of RL. Supposing at time $t$, the agent perceives the state $s^{t}$ from the environment and takes action $a^{t}$ based on the policy $\pi^{t}$. Upon taking the action, the agent receives a reward $r^{t}$, and the environmental state is updated to $s^{t+1}$. The agent repeats the above process until the terminal state $s^{t+k}$, with a sequence of rewards $\{{r^{t}},\cdots,{r^{t+k}}\}$ accrued. The key to RL is to learn the optimal policy that maximizes the cumulative reward. The Actor-Critic model is a popular algorithm for RL, in which the actor and critic models are used to learn and evaluate policies, respectively [18]. Cooperative multi-agent reinforcement learning involves multiple agents working together to achieve a coincident goal [50, 40]. Centralized learning with fully decentralized execution is one of the prominent approaches for cooperative agents [39, 38]. It entails a central control unit that observes agents’ states and regulates their actions during training. After the training phase, the central unit is removed, and each agent independently makes decisions based on the trained policies.

Individuals express their sentiment states through multiple behavioral signals, which are collected and recorded by various devices. The elements comprising each modal signal vary across modalities. For instance, text signals consist of sets of characters, while visual data is a collection of frames. Additionally, different modalities exhibit substantial variations in modal properties, noise, and even task relevance. These significant differences pose challenges for sentiment agents when uniformly dealing with multimodal signals during the modality-fusion process. Meanwhile, diverse modal properties also hinder sentiment agents from precisely extracting sentiment-related features. To provide precise sentiment signals for each sentiment agent, we propose the Modality-Sentiment Disentanglement (MSD) module to extract sentiment-related properties during the representation fission process.

Benefitting from the SAE phase, each agent perceives unimodal representations that carry sequential sentiment variations in a unified form. These sentiment properties can either be homogeneous or complementary across modalities, often referred to as modality-shared and modality-unique features. Solely extracting shared sentiment properties from each modality can be redundant, while constructing the joint representation with only complementary parts may lead to the loss of common characteristics. Thus, extracting appropriate homogeneous and complementary sentiment properties from each unimodal representation is crucial for constituting the joint representation. Yet, it also poses great challenges to related technologies in feature learning simultaneously. Cooperative control pre-defines the communication mode between agents according to the task and adjusts to the optimal policies through rewards. This working mechanism aligns well with our goal of coordinating multiple signals to achieve the optimal joint representation. Inspired by cooperative control, we propose the concept of sentiment agents and realize the interaction between multimodal signals through agents’ collaborations to finish the MRL task.

We assign sentiment agents for each modality, where unimodal representation $f_{i}$ is the input for each agent. Each sentiment agent has an independent policy model, also referred to as the actor model. It takes actions $w_{i}$ according to the current policy $\pi_{i}$ to determine valuable properties within the modality. Then, Co-SA combines selected features from all modalities into $f=({f_{v}}\times{w_{v}})*({f_{a}}\times{w_{a}})*({f_{t}}\times{w_{t}})$ and further applies it to downstream tasks, where $*$ indicates the integrated mode. The key to the SAC phase is exactly hitting features that enhance or complement across modalities through $w_{i}$, avoiding the loss or redundancy of sentiment properties.

To this end, we carefully designed policy models and their joint optimization strategy to facilitate efficient interactions across modalities. On the one hand, we allow each sentiment agent to perceive its dominant modality as well as its differential features with other modalities. On the other hand, we jointly evaluate all policy models to coordinate captured sentiment properties across modalities. Specifically, in each stage, policy models take their dominant modal representations and different features with other modalities as input for learning common and complementary components, respectively. Taking the visual modality as an example, the visual policy model takes visual representation $f_{v}$ as well as its differential features with the other two modalities $f_{va}$, $f_{vt}$ as input and output the weight $W_{v}$.

Where ${f_{va}}={F_{t}}({f_{v}}-{f_{a}};{\theta_{t}})$ and ${f_{vt}}={F_{t}}({f_{v}}-{f_{t}};{\theta_{t}})$ are differential features. Co-SA concatenates $f_{v}$, $f_{va}$, and $f_{vt}$ and then map to get action $w_{v}$. Policy models are required to be efficient on all samples. Thereby, upon outputting weights for the current batch of samples, sentiment agents transfer to the next batch of representations ${f_{i}}={f_{i}}^{\prime}\times{w_{i}}$ and exploit cumulative rewards to evaluate policy models. During the representation transition, ${f_{i}}^{\prime}$ is determined by the sampling order of the training samples. Meanwhile, Co-SA evaluates policy models according to the succession of weights they learned. It designs transformer-based critic models that learn cumulative rewards from the concatenation between current representations $f_{i}$ and weights $w_{i}$, avoiding the massive computation from the current to terminal training batches. Notably, all agents struggle to adjust policies and reach the consistent prediction. Thereby, Co-SA designs a joint critic model to output a unified cumulative reward for all policy models $Q={F_{c}}(\sum\limits_{i}^{\{v,a,t\}}{{f_{i}}\oplus{w_{i}}};{\theta_{c}})$. The joint critic model works as the central control unit that coordinates sentiment agents in continuously adjusting policies and learning appropriate cross-modal features for the joint representation. It will be removed after training, leaving policy models to decide valuable properties independently.

Clearly, policy models wish to output weights that cause a larger cumulative reward $Q$. Thereby, the objective function of policy models $L_{actor}$ is optimizing their parameters ${\theta_{\pi}}$ that maximize the output of the critic model. As for the optimization of the critic model, the Temporal-Difference (TD) Error algorithm [19] is employed: ${L_{critic}}=Q{\rm{-}}\bar{Q}$, where $\bar{Q}=r+\gamma Q^{\prime}$. $Q^{\prime}$ is the cumulative reward in the next stage, and $\gamma$ is the discounted factor. $r$ is the immediate reward, and it varies with tasks. For the MSA task, $r=-\left|{y^{\prime}-y}\right|$, where $y^{\prime}$ and $y$ are predicted and true sentiment state. For the MER task, $r=\frac{{{e^{{y_{i}}}}}}{{\sum\limits_{j}^{\rm{C}}{{e^{{y_{j}}}}}}}$, where $y_{i}$ is the predicted probability for the true class, and $C$ is the number of emotion categories. Besides, the calculation of the reward $r$ relies on both the actor-critic and prediction models. To train the model stably, Co-SA iteratively optimizes the prediction and actor-critic model parameters. Thus, Co-SA contains loss functions about the policy, critic, and prediction models in the SAC phase. Co-SA has the total objective function after incorporating losses in the SAE phase.

$L_{p}$ is the predictive loss. It is the mean absolute loss for the MSA task and cross-entropy loss for the MER task. The weights of the four objective functions will be further discussed in the experimental part.

In this section, we evaluate each component of the Co-SA framework using the MOSI and IEMOCAP databases, which are representative datasets for the MSA and MER tasks, respectively. Beyond presenting quantitative results, we also provide and compare various visualized outcomes to offer a comprehensive analysis of Co-SA. In the subsequent sections, we delve into the investigation of the MSD and DPSR modules within the SAE phase and the SAC phase, respectively.

In this section, we compare Co-SA with several state-of-the-art methods. Co-SA produces unimodal weights that dictate the features used for fusion. Without loss of generality, we employ two classic fusion modes to obtain a joint multimodal representation: addition and concatenation operations, denoted as Co-SA(add) and Co-SA(concatenate), respectively. Comparative results on the MOSI and MOSEI databases are presented in Table II. Among the compared works, MCL has so far achieved the best results on both databases. MCL is also an approach that aims to learn high-quality multimodal representations by establishing correlations between modalities. Compared with MCL, Co-SA(add) enhances performance by 0.6%, 1.1%, 1.0%, 0.028, and 0.02 on Acc7, Acc2, F1 score, MAE, and Corr, respectively, on the MOSI database. On the MOSEI database, it improves performance by 1.2%, 0.6%, and 0.5% on Acc7, Acc2, and F1 score, respectively. The correlation (Corr) predicted by Co-SA(add) for the MOSEI database remains on par with MCL. Meanwhile, the MAE of Co-SA for the MOSEI database is slightly worse than ICCN-BERT. This generalized improvement on the MSA task is attributed to the well-designed joint representation learning process. Regarding the concatenation fusion mode, Co-SA enhances Acc7, Acc2, F1 score, MAE, and Corr by 0.3%, 0.5%, 0.4%, 0.24, and 0.21, respectively, on the MOSI database, and boosts 0.7%, 0.2%, and 0.2% on Acc7, Acc2, and F1 score, respectively, on the MOSEI database. The correlation between Co-SA’s predictions and the ground truth experiences a 0.01 drop compared with MCL. However, the overall results of both fusion modes demonstrate a stable improvement compared with state-of-the-art works.

Table III presents a comparison with several state-of-the-art (SOTA) works on the IEMOCAP database. Similar to the MSA task, MCL has achieved the best results for the MER task. Compared with MCL, Co-SA exhibits slightly inferior performance in the happy class but significantly improves the recognition rates in the neutral class. Specifically, Co-SA enhances the accuracy of the neutral emotion by 1.8% and 2.4%, respectively, when using the two fusion modes. Moreover, Co-SA also significantly improves the mean accuracy across the four emotions, outperforming all existing works.

In this section, we delve into the significance of the three modalities. Taking the MOSI database as an example, we conduct experiments with various combinations of modalities and present the results in Table IV. In Table IV, ”V”, ”A”, and ”T” denote the visual, acoustic, and text modalities, respectively. From the comparison among individual modalities, it is apparent that text data plays a pivotal role among the three modalities. This phenomenon is largely due to the fact that both the visual and acoustic modalities are constrained to use extracted features (such as AUs and MFCCs) as input because of privacy concerns, whereas raw text data can be directly incorporated into Co-SA. Raw data offer a far richer source of information than extracted features. The amalgamation of text with both visual and acoustic modalities yields a noticeable improvement across all metrics, indicating that visual and acoustic signals provide effective supplementary or complementary information. Ultimately, Co-SA achieves optimal performance when all modal signals are engaged.