Denoising Bottleneck with Mutual Information Maximization
for Video Multimodal Fusion

Video multimodal fusion aims to join and comprehend information from two or more modalities in videos to make a comprehensive prediction. Early fusion model adopted simple network architectures. Zadeh et al. (2017); Liu et al. (2018a) fuse features by matrix operations; and Zadeh et al. (2018a) designed a LSTM-based model to capture both temporal and inter-modal interactions for better fusion. More recently, models influenced by prevalence of Transformer (Vaswani et al., 2017) have emerged constantly: Zhang et al. (2019) injected visual information in the decoder of Transformer by cross attention mechanism to do multimodal translation task; Wu et al. (2021) proposed a text-centric multimodal fusion shared private framework for multimodal fusion, which consists of the cross-modal prediction and sentiment regression parts. And now vision-and-language pre-training has become a promising practice to tackle video multimodal fusion tasks. (Sun et al., 2019) firstly extend the Transformer structure to video-language pretraining and used three pre-training tasks: masked language prediction, video text matching, masked video prediction.

In contrast to existing works, we focus on the fundamental characteristic of video: audio and visual inputs in video are redundant and noisy (Nagrani et al., 2021) so we aim to remove noise and redundancy while preserving critical information.

Video multimodal summarization aims to generate summaries from visual features and corresponding transcripts in videos. In contrast to unimodal summarization, some information (e.g., guitar) only exists in the visual modality. Thus, for videos, utilization of both visual and text features is necessary to generate a more comprehensive summary.

For datasets, Li et al. (2017) introduced a multimodal summarization dataset consisting of 500 videos of news articles in Chinese and English. Sanabria et al. (2018) proposed the How2 dataset consists of 2,000 hours of short instructional videos, each coming with a summary of two to three sentences.

For models, Liu et al. (2020) proposed a multistage fusion network with a fusion forget gate module, which controls the flow of redundant information between multimodal long sequences. Meanwhile, Yu et al. (2021a) firstly introduced pre-trained language models into multimodal summarization task and experimented with the optimal injection layer of visual features.

We also reduce redundancy in video like in (Yu et al., 2021a). However, we do not impair the impact of noise and redundancy in a coarse grain with forget gate. Instead, we combine fusion bottleneck and MI-Max modules to filter out noise while preserving key information.

Multimodal sentiment analysis (MSA) aims to integrate multimodal resources, such as textual, visual, and acoustic information in videos to predict varied human emotions. In contrast to unimodal sentiment analysis, utterance in the real situation sometimes contains sarcasm, which makes it hard to make accurate prediction by a single modality. In addition, information such as expression in vision and tone in acoustic help assist sentiment prediction. Yu et al. (2021b) introduced a multi-label training scheme that generates extra unimodal labels for each modality and concurrently trained with the main task. Han et al. (2021) build up a hierarchical mutual information maximization guided model to improve the fusion outcome as well as the performance in the downstream multimodal sentiment analysis task. Luo et al. (2021) propose a multi-scale fusion method to align different granularity information from multiple modalities in multimodal sentiment analysis.

Our work is fundamentally different from the above work. We do not focus on complex fusion mechanisms, but take the perspective of information in videos, and stress the importance of validity of information within fusion results.

In video multimodal fusion tasks, for each video, the input comprises three sequences of encoded features from textual ($t$), visual ($v$), and acoustic ($a$) modalities. These input features are represented as $X_{m}\in\mathbb{R}^{l_{m}\times d_{m}}$, where $m\in\{t,v,a\}$, and $l_{m}$ and $d_{m}$ denote the sequence length and feature dimension for modality $m$, respectively. The goal of DBF is to extract and integrate task-related information from these input representations to form a unified fusion result $Z\in\mathbb{R}^{l\times d}$. In this paper, we evaluate the quality of the fusion result $Z$ on two tasks: video multimodal sentiment analysis and video multimodal summarization.

For sentiment analysis, we utilize $Z$ to predict the emotional orientation of a video as a discrete category $\hat{y}$ from a predefined set of candidates $\mathcal{C}$

$$\hat{y}=\operatorname{argmax}_{y_{j}\in\mathcal{C}}\operatorname{P}_{\Theta}(y_{j}\mid Z),$$

(1)

or as a continuous intensity score $\hat{y}\in\mathbb{R}$

$$\hat{y}=\operatorname{P}_{\Theta}(Z),$$

(2)

where $\Theta$ denotes the model parameters.

For summarization, we generate a summary sequence $\hat{S}=(s_{1},...,s_{l})$ based on $Z$:

$$\hat{S}=\text{argmax}_{S}\operatorname{P}_{\Theta}(S\mid Z).$$

(3)

As shown in Figure 2, we first employ a fusion bottleneck with a restrained receptive field to perform multimodal fusion and filter out noise and redundancy in videos. Specifically, fusion bottleneck forces cross-modal information flow passes via randomly initialized bottleneck embeddings $B\in\mathbb{R}^{{l_{b}\times d_{m}}}$ with a small sequence length, where $d_{m}$ denotes dimension of features and $l_{b}\ll l$. The restrained receptive field of $B$ forces model to collate and condense unimodal information before sharing it with the other modalities.

With a small length $l_{b}$, embedding $B$ acts like a bottleneck in cross-modal interaction. In the fusion bottleneck module, unimodal features cannot directly attend to each other and they can only attend to the bottleneck embeddings $B$ to exchange information in it. Meanwhile, the bottleneck can attend to all of the modalities, which makes information flow across modalities must pass through the bottleneck with a restrained receptive field. The fusion bottleneck module forces the model to condense and collate information and filter out noise and redundancy.

Specifically, in the fusion bottleneck module, with bottleneck embeddings $B$ and unimodal features $X_{m}$, the fusion result is calculated as follows:

$$[X_{m}^{l+1}||B_{m}^{l+1}]=\text{Transformer}([X_{m}^{l}||B^{l}]),$$

(4)

$$B^{l+1}=\text{Mean}(B_{m}^{l+1}),$$

(5)

where $l$ denotes the layer number and $||$ denotes the concatenation operation. As shown in Equation 4 and 5, each time a Transformer layer is passed, bottleneck embedding $B$ is updated by unimodal features. In turn, unimodal features integrate condensed information from other modalities through bottleneck embeddings $B$. Finally, we output the text features $X_{t}^{L}$ of the last layer $L$, which are injected with condensed visual and audio information, as the fusion result.

The fusion bottleneck module constrains information flow across modalities in order to filter out noise and redundancy. However, it may result in loss of critical information as well when fusion bottleneck selects what information to be shared. To alleviate this issue, we employ a mutual information maximization (MI-Max) module to preserve representative and salient information from redundant modalities in fusion results.

Mutual information is a concept from information theory that estimates the relationship between pairs of variables. Through prompting the mutual information between fusion results $Z$ and multimodal inputs $X_{m}$, we can capture modality-invariant cues among modalities (Han et al., 2021) and keep key information preserved by regulating the fusion bottleneck module.

Since direct maximization of mutual information for continuous and high-dimensional variables is intractable (Belghazi et al., 2018), we instead minimize the lower bound of mutual information as Han et al. (2021) and Oord et al. (2018). To be specific, we first construct an opposite path from $Z$ to predict $X_{m}$ by an MLP $\mathcal{F}$. Then, to gauge correlation between the prediction and $X_{m}$, we use a normalized similarity function as follows:

$$\text{sim}(X_{m},Z)=\text{exp}\left(\frac{X_{m}}{\left\|X_{m}\right\|^{2}}\odot\frac{\mathcal{F}(Z)}{\left\|\mathcal{F}(Z)\right\|^{2}}\right),$$

(6)

where $\mathcal{F}$ generates a prediction of $X_{m}$ from $Z$, $\|\cdot\|^{2}$ is the Euclidean norm, and $\odot$ denotes element-wise product. Then, we incorporate this similarity function into the noise-contrastive estimation framework (Gutmann and Hyvärinen, 2010) and produce an InfoNCE loss (Oord et al., 2018) which reflects the lower bound of the mutual information:

$$\mathcal{L}_{\text{NCE}}^{z,m}=-\mathbb{E}_{X_{m},Z}\left[\log\frac{e^{\operatorname{sim}\left(x_{m}^{{+}},\mathcal{F}(Z)\right)}}{\sum_{k=1}^{K}e^{\operatorname{sim}\left(\tilde{x}_{m}^{k},\mathcal{F}(Z)\right)}}\right]$$

(7)

where $\tilde{x}_{m}=\left\{\tilde{x}^{1},\ldots,\tilde{x}^{K}\right\}$ is the negative unimodal inputs that are not matched to the fusion result $Z$ in same batch. Finally, we compute loss for all modalities as follows:

$$\mathcal{L}_{\text{NCE}}=\alpha(\mathcal{L}_{\text{NCE}}^{z,v}+\mathcal{L}_{\text{NCE}}^{z,a}+\mathcal{L}_{\text{NCE}}^{z,t})$$

(8)

where $\alpha$ is a hyper-parameter that controls the impact of MI-Max.

By minimizing $\mathcal{L}_{\text{NCE}}$, on the one hand, we maximize the lower bound of the mutual information between fusion results and unimodal inputs; on the other hand, we encourage fusion results to reversely predict unimodal inputs as well as possible, which prompts retaining of representative and key information from different modalities in fusion results.

We evaluate fusion results of DBF on two video multimodal tasks: video multimodal sentiment analysis and video multimodal summarization.

For sentiment analysis task, we use BERT-base (Devlin et al., 2018) to encode text input and extract the [CLS] embedding from the last layer. For acoustic and vision, we use COVAREP (Degottex et al., 2014) and Facet ¹¹1https://imotions.com/platform/ to extract audio and facial expression features. The visual feature dimensions are 47 for MOSI, 35 for MOSEI, and the audio feature dimensions are 74 for both MOSI and MOSEI.

For summarization, we use BART (Lewis et al., 2019) as the feature extractor and inject visual information in the last layer of the BART encoder. For vision, a 2048-dimensional feature representation is extracted for every 16 non-overlapping frames using a 3D ResNeXt-101 model (Hara et al., 2018), which is pre-trained on the Kinetics dataset (Kay et al., 2017). Details of the hyper-parameters are given in Appendix A. For frameworks and hardware, we use the deep learning framework PyTorch (Paszke et al., 2017) and Huggingface ²²2https://huggingface.co/ to implement our code. We use a single Nvidia GeForce A40 GPU for sentiment analysis experiments and two for summarization.

We compare performance against DBF by considering various baselines as below: For multimodal sentiment analysis, we compare with MulT (Tsai et al., 2019), TFN (Zadeh et al., 2017), LMF (Liu et al., 2018b), MFM (Tsai et al., 2018), ICCN (Sun et al., 2020), MISA (Hazarika et al., 2020), Self-MM (Yu et al., 2021b) and MMIM (Han et al., 2021). For multimodal summarization, we compare with HA (Palaskar et al., 2019) MFFG (Liu et al., 2020) VG-GPLMs (Yu et al., 2021a). Details of baselines are in Appendix B. The comparative results for sentiment analysis are presented in Table 1 (MOSI) and Table 2 (MOSEI). Results for summarization are presented in Table 3 (How2).

We find that DBF yields better or comparable results to state-of-the-art methods. To elaborate, DBF significantly outperforms state-of-the-art in all metrics on How2 and in most of metrics on MOSI and MOSEI. For other metrics, DBF achieves very closed performance to state-of-the-art. These outcomes preliminarily demonstrate the efficacy of our method in video multimodal fusion.

From the results, we can observe that our model achieves more significant performance improvement on summary task than sentiment analysis. There could be two reasons for this: 1) the size of two datasets is small, yet DBF requires a sufficient amount of data to learn noise and redundancy patterns for this type of video. 2) Visual features are extracted by Facet on sentiment analysis task and more 3D ResNeXt-101 on summary task respectively. Compared to sentiment analysis task, summary task employ a more advanced visual extractor and DBF is heavily influenced by the quality of visual features.

In this section, we first calculate standard deviation and normalized entropy over visual attention scores in the Grad-CAM heatmaps (Selvaraju et al., 2017) for DBF and baseline method VG-GPLMs (Yu et al., 2021a) respectively. These two metrics show the sharpness of visual attention scores, indicating whether the model focuses more on key frames and ignores redundant content. Then, we compute visualizations on Grad-CAM heatmaps acquired before to show the ability of DBF to filter out redundancy and preserve key information.