Mutual Information-based Representations Disentanglement for Unaligned Multimodal Language Sequences

Tensor-based methods Although tensor-based methods were initially applied to aligned multimodal language sequences, they can be easily extended to the case of unaligned data. TFN [15] utilizes an outer product-based calculation of representations encoded from each modality to capture uni-modal, bi-modal, and tri-modal interactions. LMF [16] builds upon TFN by incorporating a low-rank multimodal tensor fusion technique, enhancing the efficiency of the model.

Graph-based methods Multimodal Graph [17] is structured hierarchically for two stages: intra- and inter-modal dynamics learning. In the first stage, a graph convolutional network is used for each modality to learn intra-modal dynamics. In the second stage, a graph pooling fusion network is used to automatically learn associations between nodes from different modalities and form a multimodal joint representation. GraphCAGE [18] leverages the dynamic routing of capsule network and self-attention to create nodes and edges for each modality, which effectively address long-range dependency issues. MTGAT [19] presents a procedure for transforming unaligned multimodal sequence data into a single graph comprising heterogeneous nodes and edges. This graph effectively captures the rich interactions among diverse modalities through time. Subsequently, a novel multimodal temporal graph attention is introduced to incorporate a dynamic pruning and read-out technique, enabling the refined joint representation of this multimodal temporal graph.

Cross-modal attention-based methods MulT [20] utilizes a cross-modal attention mechanism, which provides a latent cross-modal adaptation that fuses multimodal information by directly attending to low-level features in other modalities, regardless of the need for alignment. PMR [21] utilizes a message hub to facilitate information exchange among unaligned multimodal sequences. The message hub sends common messages to each modality and reinforces their features through cross-modal attention. In turn, it gathers the reinforced features from each modality to create a reinforced common message. Through iterative cycles, the common message and modality features progressively complement each other. LMR-CBT [22] introduces a novel cross-modal block transformer, enabling complementary learning across different modalities. It comprises local temporal learning, cross-modal feature fusion, and global self-attention representations.

MISA [23] projects each modality into two subspaces: one is modality-invariant subspace for capturing commonalities and reducing modality gaps, and another is modality-specific subspace for preserving unique characteristics. These representations are fused into a multimodal joint representation for sentiment prediction. FDMER [24] handles modality heterogeneity by learning two distinct representations for each modality on aligned multimodal sequences. The first representation is the common representation for projecting all modalities into a modality-invariant shared subspace with aligned distributions. The representation can capture commonality among modalities regarding suggested emotions and reduces the modality gap. The second representation is the private representation for providing a modality-specific subspace for each modality. In this subspace, FDMER learns the unique characteristics of different modalities and eliminates redundant information. MFSA [25] employs a predictive self-attention module to capture reliable contextual dependencies and enhance unique features within modality-specific spaces. In addition, a hierarchical cross-modal attention module is introduced to explore correlations between cross-modal elements across the modality-agnostic space. To ensure the acquisition of distinct representations, a double-discriminator strategy is employed in an adversarial manner. Finally, modality-specific and modality-agnostic representations are integrated into a multimodal joint representation for downstream tasks.

Our task is defined as follows: Given a set of three modalities $M=\{L({\rm Language}),A({\rm Audio}),V({\rm Visual})\}$, an opinion video, i.e., multimodal language sequence, can be represented as $\{\mathbf{X}^{V},\mathbf{X}^{L},\mathbf{X}^{A}\}$ where $\mathbf{X}^{m}=[\mathbf{x}_{1}^{m},\mathbf{x}_{2}^{m},\\$$...,\mathbf{x}_{T_{m}}^{m}]\in\mathbb{R}^{T_{m}\times d_{m}}$, $m\in M-\{L\}$, $\mathbf{x}_{i}^{m}\in\mathbb{R}^{d_{m}}$ denotes the extracted sentiment features corresponding to modality $m$ at the $i$-th moment, $d_{m}$ is the dimension of the feature, and $T_{m}$ is the length of sequence of modality $m$. $\mathbf{X}^{L}=\left[w_{1},w_{2},...,w_{T_{L}}\right]$ is the original sequence of words. Our goal is to predict the sentiment intensity $y\in\mathbb{R}$ of the whole video.

To extract modality-specific (a.k.a., private) information, we design the unimodal encoders to map the feature sequences of each modality to their respective modality-specific subspaces, obtaining mutually exclusive modality-specific representations,

where $m\in M$, $f_{\theta_{\rm enc}^{m}}$ represents the encoder for modality $m$, $\theta_{\rm enc}^{m}$ denotes the parameter of the encoder, $\mathbf{z}^{m}\in\mathbb{R}^{d}$ is the acquired modality-specific representation, and $d$ is the dimension of the representation.

For all modalities, the Long Short-Term Memory (LSTM) [26] and subsequent fully connected networks are selected as the modality-specific encoders $f_{\theta_{\rm enc}^{L}},f_{\theta_{\rm enc}^{A}},f_{\theta_{\rm enc}^{V}}$ to model the intra-modal interactions within the sequences.

The aim of the multimodal encoder is to obtain a modality-agnostic representation from the feature sequences of three modalities, and the learning of such a representation requires cross-modal interactions. Given the outstanding performance and popularity of text-centric cross-modal interaction models, we introduce our previous work Word-wise Sparse Attention BERT (WSA-BERT) [27], which takes into account long-range dependencies across modalities and sparse attention mechanisms between sequence elements. By injecting the non-verbal (audio and visual) information into the word representations at the intermediate stage of the text backbone model, more refined word representations are obtained.

More specifically, we first feed $\mathbf{X}^{L}$ into a pretrained BERT [28] model and extract the middle word representations $\mathbf{W}^{\rm mid}=\left[\mathbf{w}_{1},\mathbf{w}_{2},...,\mathbf{w}_{T_{\scriptscriptstyle L}}\right]\in\mathbb{R}^{T_{L}\times d_{w}}$, where $\mathbf{w}_{k}\in\mathbb{R}^{d_{w}}\left(k=1,2,..,T_{L}\right)$ is the intermediate word representation at the $k$-th moment, $d_{w}$ is the dimension of the representation, and $T_{L}$ represents the length of the sequence. Then, the non-verbal feature sequences $\mathbf{X}^{A}$ and $\mathbf{X}^{V}$, along with the word representation sequence $\mathbf{W}^{\rm mid}$, are jointly mapped to a common semantic space, where the word $\mathbf{w}_{k}$ is used as a semantic anchor to search for the non-verbal information most relevant to $\mathbf{w}_{k}$ from holistic non-verbal sequences. The semantic similarity between each word representation and the non-verbal features is calculated and then followed by a Sparsemax [29] function to highlight the important information in non-verbal sequences,

where $m\in M-\{L\}$, $f_{\theta_{\rm sim}^{m}}$ is the similarity function (fully connected network and inner product similarity) with the parameter $\theta_{\rm sim}^{m}$ for modality $m$, ${\rm Sparsemax}(\cdot)$ is the Sparsemax transformation which can map attention scores to a probability simplex, $\widetilde{\mathbf{X}}^{A}\in\mathbb{R}^{T_{L}\times d_{A}}$ and $\widetilde{\mathbf{X}}^{V}\in\mathbb{R}^{T_{L}\times d_{V}}$ denote the combined audio and visual feature sequences with the same length as the language sequence $\mathbf{W}^{\rm mid}$, respectively.

Furthermore, a Multimodal Adaptive Gating (MAG) [30] mechanism is leveraged to determine the amount of information injected from non-verbal modalities. Specifically, two adaptive gates are constructed to control audio and visual information, respectively,

where $m\in M-\{L\}$, $\left[\mathbf{W}^{\rm mid};\widetilde{\mathbf{X}}^{m}\right]$ represents the concatenation of $\mathbf{W}^{\rm mid}$ and $\widetilde{\mathbf{X}}^{m}$, $\sigma$ represents a sigmoid activation function, $f_{\theta_{\rm ada}^{m}}$ is the single-layer fully connected network with the parameter $\theta_{\rm ada}^{m}$, and $\mathbf{G}^{m}\in\mathbb{R}^{T_{L}\times d_{w}}$ is the acquired adaptive gate.

Then, the sequence of displacement vectors is acquired by fusing together audio and visual features multiplied by their respective gate vectors,

where $\odot$ represents the Hadamard product (element-wise product), $f_{\theta_{\rm dis}^{V}}$ and $f_{\theta_{\rm dis}^{A}}$ are the single-layer fully connected networks with the parameters $\theta_{\rm dis}^{V}$ and $\theta_{\rm dis}^{A}$, respectively, $\mathbf{H}\in\mathbb{R}^{T_{L}\times d_{w}}$ is the acquired sequence of displacement vectors. Subsequently, the weighted summation is utilized between BERT word representations $\mathbf{W}^{\rm mid}$ and its non-verbal displacement vectors $\mathbf{H}$ to create a more refined word representations sequence containing non-verbal information,

where $\epsilon$ is a hyper-parameter, $\lambda_{k}$ is the scaling factor which designed to remain the effect of non-verbal displacement vectors $\mathbf{h}_{k}$ within a desirable range, ${\rm Diag}(\cdot)$ is a function that transforms vectors into diagonal matrices, $\widetilde{\mathbf{W}}^{\rm mid}\in\mathbb{R}^{T_{L}\times d_{w}}$ is the modified word representation incorporating global non-verbal information. We input $\widetilde{\mathbf{W}}^{\rm mid}$ into the subsequent BERT model to obtain the final word representations $\mathbf{W}^{\rm last}\in\mathbb{R}^{T_{L}\times d_{w}}$. Finally, the word representations $\mathbf{W}^{\rm last}$ are pooled and mapped to the modality-agnostic subspace,

where $\rm Pooling\left(\cdot\right)$ is the average pooling function, $f_{\theta_{\rm FC}}$ denotes the fully connected network with the parameter $\theta_{\rm FC}$, $\mathbf{z}^{S}\in\mathbb{R}^{d}$ is the acquired modality-agnostic representation, and $d$ is the dimension of the representation. For simplicity, we define the parameter of the multimodal encoder as $\theta_{\rm enc}^{S}=\{\theta_{\rm sim}^{V},\theta_{\rm sim}^{A},\theta_{\rm ada}^{V},\theta_{\rm ada}^{A},\theta_{\rm dis}^{V},\theta_{\rm dis}^{A},\theta_{\rm FC}\}$.

To ensure that the disentangled representations do not lose information and capture the underlying structure of multimodal data [31], the disentangled modality-specific representation $\mathbf{z}^{m}$ and modality-agnostic representation $\mathbf{z}^{S}$ are used to reconstruct the original representations $\mathbf{X}^{m}$,

where $m\in M$, $f_{\theta_{\rm dec}^{m}}$ is the LSTM [26] decoder for modality $m$ with the parameter $\theta_{\rm dec}^{m}$, $\widehat{\mathbf{X}}^{m}\in\mathbb{R}^{T_{m}\times d_{m}}$ is the reconstructed sequence of representations.

The representations disentanglement aims to decompose the original representation $\mathbf{X}^{m}$ into $\mathbf{z}^{m}$ and $\mathbf{z}^{S}$ $(m\in M)$, such that $\mathbf{z}^{m}$ and $\mathbf{z}^{S}$ contain completely different information. This requires removing any correlation between $\mathbf{z}^{m}$ and $\mathbf{z}^{S}$, between $\mathbf{z}^{m_{1}}$ and $\mathbf{z}^{m_{2}}$ $(m_{1},m_{2}\in M,m_{1}\neq m_{2})$. Previous representations disentanglement-based methods [23, 25] have often utilized Orthogonal Constraints (OC), taking $\mathbf{z}^{m}$ and $\mathbf{z}^{S}$ as examples, as shown below:

where $m\in M$, $\mathbf{Z}^{m}\in\mathbb{R}^{N_{\rm oc}\times d}$ and $\mathbf{Z}^{S}\in\mathbb{R}^{N_{\rm oc}\times d}$ are sets of $\mathbf{z}^{m}$ and $\mathbf{z}^{S}$ from $N_{\rm oc}$ samples, respectively, $\left\|\right\|_{F}$ represents the Frobenius norm.

The application of orthogonal constraints ensures linear independence among variables. Nonetheless, nonlinear relationships may still persist among these variables, leading to an inadequate level of representations disentanglement. To this end, we propose a constraint based on mutual information minimization, which effectively eliminates the dependency between variables. Specifically, we utilize Contrastive Log-ratio Upper Bound (CLUB) [32] to estimate the mutual information upper bound. CLUB bridges mutual information estimation with contrastive learning, where mutual information is estimated by the difference of conditional probabilities between the joint distribution (positive samples) and the marginal distribution (negative samples),

With sample pairs $\{(\mathbf{z}_{i}^{m},\mathbf{z}_{i}^{S})\}_{i=1}^{N_{\rm mi}}$, $\mathcal{I}\left(\mathbf{z}^{m},\mathbf{z}^{S}\right)$ has an unbiased estimation as:

Since the conditional distribution $p(\mathbf{z}^{S}|\mathbf{z}^{m})$ is unavailable, the above formula is difficult to calculate. Therefore, a variational distribution $q_{\theta_{\rm var}^{(m,S)}}(\mathbf{z}^{S}|\mathbf{z}^{m})$ with parameter $\theta_{\rm var}^{(m,S)}$ is used to approximate $p(\mathbf{z}^{S}|\mathbf{z}^{m})$,

In theory, as long as the variational distribution $q_{\theta_{\rm var}^{(m,S)}}(\mathbf{z}^{S}|\mathbf{z}^{m})$ is good enough, (12) remains an upper bound on the mutual information [32]. We employ the neural networks $f_{\theta_{\rm var}^{(m,S)}}$ parameterized by $\theta_{\rm var}^{(m,S)}$ to characterize the variational distribution $q_{\theta_{\rm var}^{(m,S)}}(\mathbf{z}^{S}|\mathbf{z}^{m})$, and by enhancing the capacity of the neural network and updating its parameters, we can obtain a more accurate approximation of $p(\mathbf{z}^{S}|\mathbf{z}^{m})$. Thus, the neural networks are referred to as the mutual information estimators.

To minimize the mutual information between $\mathbf{z}^{m}$ and $\mathbf{z}^{S}$, at each training iteration, we first update the variational approximation $q_{\theta_{\rm var}^{(m,S)}}(\mathbf{z}^{S}|\mathbf{z}^{m})$ by minimizing the negative log-likelihood,

After $q_{\theta_{\rm var}^{(m,S)}}(\mathbf{z}^{S}|\mathbf{z}^{m})$ is updated, we calculate the estimator as described in (12). Finally, $q_{\theta_{\rm var}^{(m,S)}}(\mathbf{z}^{S}|\mathbf{z}^{m})$ and $\left(f_{\theta_{\rm enc}^{m}},f_{\theta_{\rm enc}^{S}}\right)$ are updated alternately during the training.

In addition to enforcing the independence of information between $\mathbf{z}^{V}$, $\mathbf{z}^{L}$, $\mathbf{z}^{A}$ and $\mathbf{z}^{S}$, it is also necessary to impose constraints on the independence of information between $\mathbf{z}^{S}$ and the original inputs $\mathbf{X}^{V}$, $\mathbf{X}^{L}$, $\mathbf{X}^{A}$ to truly learn the shared information between modalities. As shown in Figure 3, taking visual and audio modalities as an example, without the constraint of information independence between $\mathbf{z}^{S}$ and the original inputs, $\mathbf{z}^{S}$ may contain excessive information beyond the genuine shared information. Thus we further use variational distribution $q_{\theta_{\rm var}^{(Xm,S)}}(\mathbf{z}^{S}|\mathbf{X}^{m})$ to minimize the mutual information between $\mathbf{z}^{S}$ and $\mathbf{X}^{m}$,

The corresponding negative log-likelihood is as follows,

For simplicity, we define the parameter of the variational distributions as $\theta_{\rm var}=\left\{\theta_{\rm var}^{(V,L)},\theta_{\rm var}^{(V,A)},\theta_{\rm var}^{(L,A)},\theta_{\rm var}^{(V,S)},\theta_{\rm var}^{(L,S)},\theta_{\rm var}^{(A,S)},\theta_{\rm var}^{(XV,S)},\theta_{\rm var}^{(XL,S)},\theta_{\rm var}^{(XA,S)}\right\}$.

After obtaining the disentangled representations $(\mathbf{z}^{V},\mathbf{z}^{L},$ $\mathbf{z}^{A},\mathbf{z}^{S})$, a regressor $f_{\theta_{\rm reg}}$ is designed to predict the sentiment intensity of the video,

where $\mathbf{p}\in\mathbb{R}^{4d}$ is the concatenation of $\mathbf{z}^{V},\mathbf{z}^{L},\mathbf{z}^{A}$ and $\mathbf{z}^{S}$, $f_{\theta_{\rm reg}}$ is a two-layer fully connected network with the parameter $\theta_{\rm reg}$, $\widehat{y}$ is the predicted sentiment intensity. For simplicity, we define $\theta=\left\{\theta_{\rm enc}^{V},\theta_{\rm enc}^{L},\theta_{\rm enc}^{A},\theta_{\rm enc}^{S},\theta_{\rm dec}^{V},\theta_{\rm dec}^{L},\theta_{\rm dec}^{A},\theta_{\rm reg}\right\}$.

The task-specifc loss estimates the quality of prediction during training. We use the Mean Squared Error (MSE) loss for the regression task,

where $N_{\rm bs}$ is the number of the samples in each batch, taking the logarithm of the MSE loss is done to avoid cumbersome weight hyper-parameter tuning and balance the gradients of each loss during the optimization process [33]. For reconstruction of original representations, the MSE loss is utilized for audio and visual modalities, and the Cross Entropy (CE) loss is utilized for language modality ( sentences are composed of discrete words in the lexicon),

Similarly, taking the logarithm of the reconstruction loss for each modality is done to automatically balance the gradients of each loss during the optimization process and avoid manual adjustment of the loss weights. Furthermore, we define $\widehat{\mathcal{I}}\left(\mathbf{z}^{m},\mathbf{z}^{S}\right)$ as $\mathcal{L}_{\rm mim}^{(m,S)}$, $\widehat{\mathcal{I}}\left(\mathbf{z}^{m_{1}},\mathbf{z}^{m_{2}}\right)$ as $\mathcal{L}_{\rm mim}^{(m_{1},m_{2})}$, $\widehat{\mathcal{I}}\left(\mathbf{X}^{m},\mathbf{z}^{S}\right)$ as $\mathcal{L}_{\rm mim}^{(Xm,S)}$ $\left(m,m_{1},m_{2}\in M\right)$, thus the Mutual Information Minimization (MIM) loss is as follows:

where $\overline{M}=M+\{S\}=\{L,A,V,S\}$, $\overline{m}_{1}\neq\overline{m}_{2}$. The overall learning of the model is performed by minimizing:

where $\alpha$ is the weight coefficient. Since the estimated mutual information values may become negative, we do not take their logarithm. Instead, we simply set the weights of the mutual information minimization losses to be the same. Meanwhile, as mentioned in Section 3.5, before optimizing $\mathcal{L}$, we need to minimize the negative log-likelihood $\mathcal{L}_{\rm lld}$ for accurate mutual information estimation,

where $\overline{M}=M+\{S\}=\{L,A,V,S\}$, $\overline{m}_{1}\neq\overline{m}_{2}$. $\mathcal{L}$ and $\mathcal{L}_{\rm lld}$ are optimized alternately during the training.

However, the estimation of mutual information may not be accurate due to limited labeled data availability [32], and introducing additional data can improve this. Thus, given the abundant supply of unlabeled opinion videos accessible on the Internet, we leverage them to assist in computing mutual information. On the other hand, these unlabeled data is also utilized to reconstruct original inputs, thereby aiding in characterizing the underlying structure of multimodal data, and further improving the performance of the model. The overall optimization algorithm of our model is shown in Algorithm 1,

We conduct the evaluation of our methods on two extensively used datasets, namely CMU-MOSI [34] and CMU-MOSEI [2], which encompass three modalities: visual, language, and audio, to convey the sentiment of the speakers. In addition, a sentiment-unlabeled dataset EmoVoxCeleb [35] is introduced for accurately estimating mutual inforamtion and reconstructing original inputs.

CMU-MOSI dataset comprises 93 opinion videos extracted from YouTube movie reviews. Each video is composed of multiple opinion segments, with each segment annotated with sentiment scores ranging from -3 (strong negative) to +3 (strong positive). To align with previous studies [20, 12, 25], we employ 1,284 segments for training, 229 segments for validation, and 686 segments for testing.

CMU-MOSEI dataset, currently the largest multimodal sentiment analysis dataset, contains 22,846 movie review video clips sourced from YouTube. Each video clip is assigned a sentiment score between -3 (strong negative) and +3 (strong positive). To align with previous studies [20, 12, 25], we utilize 16,326 video clips for training, 1,861 video clips for validation, and 4,659 video clips for testing.

EmoVoxCeleb comprises a vast collection of over 100,000 video clips featuring over 1,200 speakers, also sourced from open-source media YouTube. The dataset exhibits a balanced distribution across genders and encompasses speakers with diverse ethnicities, accents, professions, and ages. Through a rough screening process, non-English video clips were excluded, resulting in the selection of 132,708 English video clips, representing 1,105 speakers. We select a total of 81,630 video clips, which is five times the size of the training set in CMU-MOSEI, as the unlabeled data.

The sentiment-related features are extracted for non-verbal (visual and audio) modalities.

Visual: The MultiComp OpenFace2.0 toolkit [36] is employed to extract a comprehensive set of visual features, encompassing 340-dimensional facial landmarks, 35-dimensional facial action units, 6-dimensional head pose and orientation, 40-dimensional rigid and non-rigid shape parameters, and 288-dimensional eye gaze ¹¹1For more details, you can see https://github.com/TadasBaltrusaitis/OpenFace/wiki/Output-Format.

Audio: The librosa library [37] is utilized for extracting frame-level acoustic features which include 1-dimensional logarithmic fundamental frequency (log F0), 20-dimensional Mel-Frequency Cepstral Coefficients (MFCCs), and 12-dimensional Constant-Q chromatogram (CQT). These features are related to emotions and tone of speech according to [38].

All models are developed using the PyTorch toolbox [39] and trained on NVIDIA RTX 3090 GPUs. We employ the AdamW [40] optimizer with an initial learning rate of $1e-5$ for the encoders, decoders, and the regressor. For the MIM module, we utilize the Adam [41] optimizer with an initial learning rate of $1e-3$. The batch size $N_{\rm bs}$ is set to 32, and the training process runs for 100 epochs. The mutual information estimator is a two-layer fully connected network with a hidden layer dimension of 32. The number $N_{\rm mi}$ of sample pairs for mutual information estimator is equal to batch size $N_{\rm bs}$, and the number $T^{\prime}$ of iterations is set to 5. The hyper-parameter $\epsilon$ in MAG is set to 1, and the dimension $d$ of the latent representations is 64. All our experiments are conducted using the exact same random seed. In addition, we utilize the designated validation sets of CMU-MOSI and CMU-MOSEI datasets to identify the best hyper-parameters for our models.

Consistent with previous studies [20, 12, 25], we record our experimental results in two forms: classification and regression. For classification, we report the weighted F1 score and binary accuracy. For regression, we report Mean Absolute Error (MAE) and Pearson correlation (Corr). Except for MAE, higher values indicate better performance for all metrics.

The comparison results with state-of-the-art models on the CMU-MOSI and CMU-MOSEI datasets are shown in Table 1. $\uparrow$ means higher is better, and $\downarrow$ means lower is better. (B) means the language features are based on BERT [28]. ^◇ means the results are from [12]; $\S$ from original papers. ^∗ denotes the reimplementation with non-verbal sentiment features mentioned in Section 4.2. $\rhd$ denotes other representations disentanglement-based methods. MIRD¹ and MIRD² indicate the non-use and use of unlabeled data, respectively. % is omitted after Acc and F1 metrics. Best results are highlighted in bold.

From the results we can observe that, 1) our method MIRD (with or without unlabeled data) performs much better than all compared methods. The Accuracy, F1 score, MAE, and Correlation metrics of MIRD with unlabeled data are 86.23%, 86.16%, 0.721, and 0.804 on CMU-MOSI dataset, 86.34%, 86.29%, 0.553, and 0.823 on CMU-MOSEI dataset. The absolute improvements of the performance are 1.02%, 0.98%, 0.052, and 0.030 on CMU-MOSI dataset, 2.27%, 2.17%, 0.003, and 0.073 on CMU-MOSEI dataset compared to the second-best method Self-MM [12]. 2) MIRD outperforms previous representations disentanglement-based methods, MISA [23] and MFSA [25], on both datasets (we do not make comparisons with [42] and [24] since both of works are conducted on aligned multimodal language sequences, and official code implementations are not provided). These observations indicate that MIRD is more effective in modeling unaligned multimodal language sequences.

To analyze the efficiency of MIRD, we list the number of parameters (#Params) and computation (Computational Budget) of each model, as shown in Table 2. Same as Table 1, (B) means the language features are based on BERT [28]; $\rhd$ denotes other representations disentanglement-based methods. The M stands for millions; The GMACs denotes Giga Multiply–Accumulate Operations. From the table we can find that the model efficiency of MIRD is also remarkable under the premise of the best performance, especially compared with other representations disentanglement-based methods. This indicates that our method has relatively high applicability.

To illustrate the differences between our method and previous representations disentanglement-based methods, we conduct a comparison in Table 3. In the table, ✓ denote presence, while ✗ indicate absence. The difference between the multimodal encoder and the common encoder is that the input of the multimodal encoder is multiple modalities together, while the input of the common encoder is one of each modality. Our method involves learning latent representations through cross-modal interactions, followed by a straightforward concatenation, whereas other representations disentanglement-based methods directly learn latent representations and then use the Transformer [43] for fusion.

To further explore the contributions of different components, we conduct an ablation study on CMU-MOSI dataset and the results are shown in Table 4. From this table, we can observe:

1) Experiment 1 removes the labeled data and uses only labeled data. The performance decreases significantly (accuracy drops from 86.23% to 85.82%, F1 score drops from 86.16% to 85.74%), highlighting the importance of utilizing more data to estimate the mutual information and helping to characterize the underlying structure of the multimodal data. However, despite the performance degradation observed in our method without unlabeled data, comparing it with other methods in Table 1, our method still significantly outperforms the alternatives. This shows the effectiveness of the proposed disentanglement architecture along with information constraints.

2) Experiment 2, which omits the mutual information minimization constraint between the shared representation $\mathbf{z}^{S}$ and the input $\mathbf{X}^{m}$, does not result in a degradation of model performance. This is because after the representations disentanglement, there is an aggregation process on the disentangled representations, and whether or not the mutual information minimization constraint is applied does not affect the information integrity of the aggregated multimodal joint representation $\mathbf{z}$.

3) Experiment 3 removes the mutual information minimization constraints among the latent representations $\mathbf{z}^{V},\mathbf{z}^{L},\mathbf{z}^{A}$ and $\mathbf{z}^{S}$. The performance further deteriorates (accuracy drops from 85.80% to 85.21%, F1 score drops from 85.77% to 85.09%), indicating that explicitly learning modality-specific and modality-agnostic representations is beneficial for sentiment prediction.

4) Experiment 4 indicates that if the original inputs are not reconstructed and only the WSA-BERT model [27] is used for sentiment prediction, the performance of the model decreases to an accuracy of 83.99% and an F1 score of 83.83%, further demonstrating the effectiveness of the proposed method.

The utilization of Orthogonal Constraint (OC) can enhance the linear independence between variables. However, it is possible that non-linear relationships still exist among them, leading to inadequate disentanglement of representations. The representations disentanglement method based on Mutual Information Minimization (MIM) can effectively eliminate the dependencies between variables, thereby achieving a more comprehensive disentanglement of representations across different modalities. This reduces information redundancy in the multimodal joint representation and consequently enhances the performance of the model. To validate this, we compare the performance of two constrained methods and visualize their disentangled representations, as shown in Table 5 and Figure 4.

From Table 5, we can observe that the MIM-based method outperforms the OC-based method in almost all evaluation metrics on the CMU-MOSI and CMU-MOSEI datasets. Specifically, on the CMU-MOSI dataset, the MIM-based method outperforms the OC-based method by 0.45% in terms of both accuracy and F1 score; on the CMU-MOSEI dataset, the MIM-based method exhibits superior accuracy and F1 score compared to the OC-based method, with differences of 0.41% and 0.43%, respectively. From Figure 4 (a), we can observe that in the method without relationship constraints, the disentangled visual representations (orange points) are separated from the language representations (blue points), and the major portion of the visual representations spontaneously form two distinct clusters. Furthermore, the disentangled audio representations (green points) also exhibit a rough formation of two clusters. From Figure 4 (b), it can be seen that in the OC-based method, the disentangled language representations (blue points) are separated from the visual representations (orange points). However, the MIM-based method tends to roughly cluster language, visual, audio, and shared representations separately, demonstrating greater distinctiveness (as depicted in Figure 4 (c)). It suggests that imposing relationship constraints on the latent representations yields more discriminative representations, with the MIM-based method showing superior distinctiveness compared to the OC-based method. Interestingly, in Figure 4 (a), we observe that despite the absence of any relationship constraint, the intermediate representations obtained by the model still exhibit some level of separability. This could be attributed to the fact that the individual modality encoders as well as the multimodal encoder have learned certain discriminative information.

1) Mutual information estimation: The most direct reason for using unlabeled data is to estimate mutual information more accurately [32]. We plot the line chart of mutual information estimation values during the training process, as shown in Figure 5. The mutual information estimation values refer to the sum of mutual information between language, visual, audio, and shared representations pair-wise in each epoch, averaged over batches. From the figure, we can observe that without unlabeled data, the estimated mutual information converges rapidly (converging to approximately 0.4) but then exhibits intense oscillations. On the other hand, with unlabeled data, the mutual information estimation continuously decreases, approaching convergence at a smaller value 0.2, and the overall trend remains smooth. Therefore, the addition of unlabeled data can lead to more accurate mutual information estimation and more stable training process.

2) Model performance: The performance improvement resulting from the utilization of unlabeled data may be attributed to better estimating mutual information, thereby eliminating redundancy in multimodal joint representations. It may also be due to assisting in characterizing the structure of multimodal data by reconstruction, preventing model overfitting. To explore this, we conducted experiments on the CMU-MOSI dataset, and the results are shown in Table 6. In the table, \raisebox{-0.8pt}{1}⃝ refers to MIRD², which uses unlabeled data to estimate mutual information and reconstruct the original input; \raisebox{-0.8pt}{2}⃝ indicates using unlabeled data to estimate mutual information without reconstructing the original input; \raisebox{-0.8pt}{3}⃝ represents using unlabeled data to reconstruct the original input without estimating mutual information; \raisebox{-0.8pt}{4}⃝ refers to MIRD¹, which does not use any unlabeled data. Through the comparison of \raisebox{-0.8pt}{1}⃝ with \raisebox{-0.8pt}{2}⃝ and \raisebox{-0.8pt}{3}⃝, as well as \raisebox{-0.8pt}{4}⃝ with \raisebox{-0.8pt}{2}⃝ and \raisebox{-0.8pt}{3}⃝, we observe that the improvement in model performance is not only attributed to unlabeled data assisting in estimating mutual information but also to its contribution in reconstructing the original input and capturing the underlying structure of multimodal data.

In addition, to investigate the impact of the amount of unlabeled data on model performance, we present the bar chart in Figure 6. From the figure, we observe that as the amount of unlabeled data increases, the model performance improves. The model achieves its best performance (86.23% on accuracy, 86.16% on F1 score) when the split rate is equal to 3. However, further increasing the amount of unlabeled data results in a gradual decline in performance. This is because (1) the model may have already learned sufficient information from the unlabeled data, and further increasing the data does not significantly improve the performance. (2) given the limited capacity of the model, extensive unlabeled data might hinder the model from effectively utilizing all the available data. Instead, it could introduce excessive noise, potentially leading to a decline in model performance.

In order to select the optimal loss weight $\alpha$, we compare the performance of the model under different loss weights on CMU-MOSI and CMU-MOSEI dataset, as shown in Figure 7. From the figure, we can observe that on the CMU-MOSI dataset, the best performance is achieved when $\alpha$ equals to 0.1, while on the CMU-MOSEI dataset, the best performance is achieved when $\alpha$ equals to 1. This indicates that the weight of mutual information minimization should neither be too large nor too small. If it is too large, it can make the model difficult to learn, while if it is too small, it may not effectively serve as a regularization term. In addition, we find that when $\alpha$ equals to 10, the performance of the model was significantly poor, with 58.23% accuary and 58.20% F1 score on CMU-MOSI dataset, 58.08% accuracy and 58.02% F1 score on CMU-MOSEI dataset (for aesthetic reasons, we did not depict the performance when $\alpha$ equals to 10 in the figure). The reason is that the model prioritize disentangling correlations between latent representations, thereby neglecting the primary task of sentiment prediction.

To visually validate the reliability of our model, we present three examples shown in Figure 8. The examples are from the first three speaker videos of the test set of the CMU-MOSI dataset. From the figure, we can observe that the MIM-based approach exhibits the best sentiment prediction performance on the first two examples, and it is also comparable to the OC-based approach on the last example. This validates the generalizability of our proposed method in sentiment prediction tasks. Furthermore, we note that the method incorporating latent representation constraints (MIM, OC) outperforms unconstrained method (NC) in terms of sentiment prediction ability, indicating the significance of representations disentanglement.