Deep Equilibrium Multimodal Fusion

Our DEQ fusion is particularly built on deep equilibrium models to recursively capture intra- and inter-modality interactions for multimodal fusion. The traditional deep neural networks have finite depth and perform the backward pass through every layer. Two interesting observations are that the hidden layers tend to converge to some fixed points, and employing the same weight in each layer of the network, so-called weight tying, still achieves competitive results. That leads to the design principles of deep equilibrium models and the goal is to simulate an infinite depth weight-tied deep network, producing high-level and stable feature representations.

Formally, the standard DEQ [6] is formulated as a weight-tied network, and such a network with parameter $\theta$ and a depth of $L$ computes a hidden state $\mathbf{z}$ as

$$\mathbf{z}^{[j+1]}=f_{\theta}(\mathbf{z}^{[j]};\mathbf{x}),\quad j=0,\dots,L-1$$

(1)

where the untransformed input $\mathbf{x}$ is injected at each layer, $\mathbf{z}^{[j]}$ is the hidden state at layer $j$ and $\mathbf{z}^{[0]}=\mathbf{0}$. As claimed in [6], the core idea of DEQ is that when there are infinite layers ($L\rightarrow\infty$), the system tends to converge to an equilibrium state $\mathbf{z}^{*}$ such that

$$\mathbf{z}^{*}=f_{\theta}(\mathbf{z}^{*};\mathbf{x}).$$

(2)

In practice, naively computing the equilibrium state requires excessive runtime. One convergence acceleration is to formulate Eq. 2 into a root-finding problem:

$$g_{\theta}(\mathbf{z};\mathbf{x})=f_{\theta}(\mathbf{z};\mathbf{x})-\mathbf{z}.$$

(3)

Some root solvers can then be applied to the residual $g_{\theta}$ to find the equilibrium state

$$\mathbf{z}^{*}=\mathrm{RootSolver}(g_{\theta};\mathbf{x}).$$

(4)

Instead of backpropagating through each layer, we can compute gradients analytically as

$$\frac{\partial\ell}{\partial(\cdot)}=\frac{\partial\ell}{\partial\mathbf{z}^{*}}\left(\left.{-J_{g_{\theta}}^{-1}}\right|_{\mathbf{z}^{*}}\right)\frac{\partial f_{\theta}(\mathbf{z};\mathbf{x})}{\partial(\cdot)},$$

(5)

where $\ell=\mathcal{L}(\mathbf{z}^{*},\mathbf{y})$ is a loss between $\mathbf{z}^{*}$ and the target $\mathbf{y}$, $\left.{J_{g_{\theta}}^{-1}}\right|_{\mathbf{z}^{*}}$ is the inverse Jacobian of $g_{\theta}$ at $\mathbf{z}^{*}$. As it is expensive to compute the inverse Jacobian term, [6] proposed to alternatively solve a linear system by involving a vector-Jacobian product

$$\mathbf{x}\left(\left.{J_{g_{\theta}}}\right|_{\mathbf{z}^{*}}\right)+\frac{\partial\ell}{\partial\mathbf{z}^{*}}=\mathbf{0}.$$

(6)

With the formulation above, DEQ represents an infinite depth network with just one layer $f_{\theta}$, which converges to an equilibrium state, and can be backpropagated implicitly with a single computation.

Next, we formulate our DEQ fusion method. Given a set of unimodal features $\mathbf{x}=\{\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{N}\}$ from $N$ modalities, our goal is to find a unified feature that integrates the information from all modalities. To ensure the informativeness of our final integrated feature, we first execute another nonlinear projection $f_{\theta_{i}}(\cdot)$ to extract higher-level information within each modality:

$$\mathbf{z}_{i}^{[j+1]}=f_{\theta_{i}}(\mathbf{z}_{i}^{[j]};\mathbf{x}_{i}),$$

(7)

where $\mathbf{z}_{i}^{[j]}$ is the $j$-th output of the layer for modality $i$ and $\mathbf{z}_{i}^{[0]}$ is initialized to $\mathbf{0}$. $\mathbf{x}_{i}$ is the injected input feature for modality $i$. Our fusion design is flexible from the standpoint that $f_{\theta_{i}}(\cdot)$ can be altered arbitrarily to fit multiple modalities. In our case, $f_{\theta_{i}}(\cdot)$ is designed to be similar to a simple residual block [18]. Following [7], we adopt group normalization [62] instead of batch normalization [22] for stability. Hence, $f_{\theta_{i}}(\cdot)$ is formulated as

$$\begin{gathered}\hat{\mathbf{z}}_{i}^{[j]}=\mathrm{ReLU}\left(\mathrm{GroupNorm}\left(\hat{\theta}_{i}\mathbf{z}_{i}^{[j]}+\hat{\mathbf{b}}_{i}\right)\right)\\ \tilde{\mathbf{z}}_{i}^{[j]}=\mathrm{GroupNorm}\left(\tilde{\theta}_{i}\hat{\mathbf{z}}_{i}^{[j]}+\mathbf{x}_{i}+\tilde{\mathbf{b}}_{i}\right)\\ f_{\theta_{i}}(\mathbf{z}_{i}^{[j]};\mathbf{x}_{i})=\mathrm{GroupNorm}\left(\mathrm{ReLU}\left(\tilde{\mathbf{z}}_{i}^{[j]}\right)\right),\end{gathered}$$

(8)

where $\hat{\theta}_{i}$ and $\tilde{\theta}_{i}$ are the weights, $\hat{\mathbf{b}}_{i}$ and $\tilde{\mathbf{b}}_{i}$ are the bias. Given this set of modality-wise features $\{\mathbf{z}_{i}^{[j+1]}\}$ computed from $f_{\theta_{i}}(\cdot)$, where $i=1,2,\dots,N$, our target is to fuse them to obtain a unified feature integrating the information from all $N$ modalities. In addition, considering that the dimension of this unified feature is limited, it necessitates dynamically selecting the most representative information from each modality-wise feature to reduce redundancy.

We propose a dynamic purify-then-combine fusion strategy for this purpose. We account for feature correlation between the fused feature and the modality-wise features by applying a soft gating function $G(\cdot)$, to dynamically model feature correlation via computing a weight $\alpha_{i}$ for each modality:

$$\begin{gathered}\alpha_{i}=G(\mathbf{z}_{\mathrm{fuse}}^{[j]},\mathbf{z}_{i}^{[j+1]})\\ G(\mathbf{z}_{\mathrm{fuse}}^{[j]},\mathbf{z}_{i}^{[j+1]})={\theta_{\alpha}}\left(\mathbf{z}_{\mathrm{fuse}}^{[j]}+\mathbf{z}_{i}^{[j+1]}\right)+{\mathbf{b}_{\alpha}},\end{gathered}$$

(9)

where $\mathbf{z}_{\mathrm{fuse}}^{[j]}$ is the fused feature from the $j$-th layer and $\mathbf{z}_{\mathrm{fuse}}^{[0]}$ is initialized to $\mathbf{0}$. $\theta_{\alpha}$ and $\mathbf{b}_{\alpha}$ are the weight and bias. The gating function $G(\cdot)$ assigns the larger weights to parts of the fused feature that better encode the information from modality $i$. We purify the fused feature with the correlation weight for modality $i$:

$$\mathbf{z}_{i}^{\prime}=\alpha_{i}\odot\mathbf{z}_{\mathrm{fuse}}^{[j]},$$

(10)

where $\odot$ represents element-wise multiplication. $\mathbf{z}_{i}^{\prime}$ could be interpreted as the significant feature purified from the fused feature that represents the information of modality $i$ from previous layers. We then combine these purified features and adopt a simplified residual block to obtain the unified feature as

$$\begin{gathered}\hat{\mathbf{z}}_{\mathrm{fuse}}={\theta}_{\mathrm{fuse}}\cdot\sum_{i=1}^{N}\mathbf{z}_{i}^{\prime}+{\mathbf{b}}_{\mathrm{fuse}}\\ \mathbf{z}_{\mathrm{fuse}}^{[j+1]}=\mathrm{GroupNorm}\left(\mathrm{ReLU}\left(\hat{\mathbf{z}}_{\mathrm{fuse}}+\mathbf{x}_{\mathrm{fuse}}\right)\right),\end{gathered}$$

(11)

where $\mathbf{x}_{\mathrm{fuse}}$ is the injected input fused feature computed from the set of modality-wise features $\{\mathbf{x}_{i}\}$ for $i=1,2,\dots,N$, $\theta_{\mathrm{fuse}}$ and $\mathbf{b}_{\mathrm{fuse}}$ are the weight and bias. In shallow layers (small $j$), $\mathbf{z}_{\mathrm{fuse}}^{[j]}$ encodes low-level modality interactions. As we continuously summarize the purified feature $\mathbf{z}_{i}^{\prime}$, i.e., $j$ gets larger and larger, $\mathbf{z}_{\mathrm{fuse}}^{[j]}$ tends to capture higher-level modality interactions while recursively integrating low-level information from previous iterations. By doing so, the final $\mathbf{z}_{\mathrm{fuse}}^{[\infty]}$ integrates the cross-modality interactions and correlations ranging from low level to high level. Moreover, our approach is flexible on the ways to compute the injected fused feature $\mathbf{x}_{\mathrm{fuse}}$. In our case, we compute it with a simple weighted sum:

$$\mathbf{x}_{\mathrm{fuse}}=\sum_{i=1}^{N}w_{i}\mathbf{x}_{i},$$

(12)

where $w_{i}$ is a learnable weight associated with modality $i$ representing modality importance.

We denote the above-proposed fusion module in Eqs. 9, 10 and 11 as a nonlinear function $f_{\mathrm{fuse}}(\cdot)$ such that

$$\mathbf{z}_{\mathrm{fuse}}^{[j+1]}=f_{\mathrm{fuse}}(\mathbf{z}_{\mathrm{fuse}}^{[j]};\mathbf{x}),$$

(13)

where $\mathbf{x}=\{\mathbf{x}_{i}\}$ for $i=1,2,\dots,N$ is the set of the injected modality-wise features. Ideally, a superior unified feature should capture the information from all modalities at every level and thus we progressively model modality interactions from low-level to high-level feature space. Technically, we present to recursively interchange intra- and inter-modality information until the equilibrium state is reached, to obtain such an informative unified representation in a stable feature space for multimodal learning. To achieve this goal, we leverage the idea of DEQs into our multimodal fusion framework. Considering $f_{\theta_{i}}(\cdot)$ for $i=1,2,\dots,N$ and $f_{\mathrm{fuse}}(\cdot)$ as DEQ layers, we aim to find equilibrium states such that

$$\mathbf{z}_{i}^{*}=f_{\theta_{i}}\left(\mathbf{z}_{i}^{*};\mathbf{x}_{i}\right),\quad\mathbf{z}_{\mathrm{fuse}}^{*}=f_{\mathrm{fuse}}\left(\mathbf{z}_{\mathrm{fuse}}^{*};\mathbf{x}\right),$$

(14)

where $\mathbf{z}_{\mathrm{fuse}}^{*}$ and $\mathbf{z}_{i}^{*}$, $i=1,2,\dots,N$, are the fused feature and all unimodal features in equilibrium states respectively. Note that we also keep track of computation for each unique modality-wise feature, so that the information from different modalities can be aligned and captured at a stable level together with the fused feature. We conduct ablation studies to demonstrate the superiority of our purify-then-combine fusion strategy compared to other fusion variants involving DEQs. Please refer to Section 4.2 for more details.

The fixed points in Eq. 14 can be reformulated into residual functions for the root-finding problem:

$$g_{\theta_{i}}(\mathbf{z}_{i};\mathbf{x}_{i})=f_{\theta_{i}}(\mathbf{z}_{i};\mathbf{x}_{i})-\mathbf{z}_{i},$$

(15)

$$g_{\mathrm{fuse}}(\mathbf{z}_{\mathrm{fuse}};\mathbf{x})=f_{\mathrm{fuse}}(\mathbf{z}_{\mathrm{fuse}};\mathbf{x})-\mathbf{z}_{\mathrm{fuse}}$$

(16)

Finally, we can solve for features in equilibrium states via a black-box solver by minimizing the residuals $g_{\theta_{i}}$ for $i=1,2,\dots,N$ and $g_{\mathrm{fuse}}$:

$$\mathbf{z}^{*},\mathbf{z}_{\mathrm{fuse}}^{*}=\mathrm{RootSolver}(g_{\theta};g_{\mathrm{fuse}};\mathbf{x}),$$

(17)

where $\mathbf{z}^{*}=\{\mathbf{z}_{i}^{*}\}$ and $g_{\theta}=\{g_{\theta_{i}}\}$ for $i=1,2,\dots,N$. Fig. 1 illustrates an overview of our deep equilibrium fusion architecture.

A benefit of using DEQs compared to stacking conventional networks is that the gradients can be computed analytically without tracing through the forward pass layer-by-layer.

The proof for Theorem 1 is provided in Appendix A. The gradients with respect to parameters of DEQ layers can be computed following Eq. 5.

BRCA. We compare our DEQ fusion approach with several baseline fusion methods, including the best competitor MM-Dynamics [16], in Table 1. It is noticeable that the complementarity of some modalities is significant, as approximately -10% performance drop is observed without mRNA data. This also somewhat manifests the advantage of dynamic modeling to take multiple modality signals into account. Similar to our dynamic design with a soft gating function, MM-Dynamics models feature and modality informativeness dynamically for trustworthy multimodal fusion. Our DEQ fusion additionally considers intra- and inter-modality features at every level, outperforming MM-Dynamics in all evaluation metrics. Notably, our method with two modalities of mRNA and DNA methylation already attains better performance in all evaluation metrics compared to MM-Dynamics which leverages all three modalities. All above results demonstrate the effectiveness of capturing modality interactions ranging from low level to high level in our deep equilibrium fusion design.

MM-IMDB. We compare our DEQ fusion strategy with various baseline fusion methods in Table 2. It is clear that text modality is more representative than image modality for this classification task, as unimodal text models exhibit significantly better performance than unimodal image models. As such, existing approaches which do not involve dynamic modeling of modality information, attain either similar performance or minor improvement compared to the unimodal text baseline. A dynamic fusion strategy is seemingly crucial to further leverage the information from the relatively weak image signal for better performance. DynMM [64] capitalizes on hard gating to select the most appropriate fusion strategy from a set of predefined operations to achieve better results. We experiment with a late fusion strategy by simply replacing the original concatenation fusion with our DEQ fusion module. With this simple modification, we obtain the state-of-the-art results of 61.52% and 53.38% for micro and macro F1 scores respectively on MM-IMDB benchmark, which is a significant improvement of 2.50% and 3.11% against the late fusion baseline, also 1.17% and 1.78% improvement compared to DynMM.

CMU-MOSI. We compare our fusion approach with several baseline fusion methods, including the state-of-the-art CM-BERT [65], in Table 3. It is worth noting that BERT-based methods exhibit better performance than other baseline approaches. For instance, vanilla BERT [12], leveraging only text modality, already surpasses other non-BERT methods which involve the utilization of all three modalities. We speculate that text modality provides more significant information for sentiment analysis task than the other two modalities. CM-BERT exploits audio modality in addition to BERT for further performance boost. Our DEQ fusion benefits from the dynamic and stable modality information modeling, and interaction exchange at every level with our recursive fusion design, outperforming CM-BERT by 1.2%, 0.9%, and 0.9% in Acc7, Acc2, and F1 score, respectively.

SUN RGB-D. We report mean Average Precision (mAP) with 3D IoU thresholds of 0.25 and 0.5 measured on multiple 3D object detection methods in Table 4. Interestingly, adding the additional RGB modality without advanced fusion mechanism harms the performance, e.g., including RGB modality into GroupFree [32] and VoteNet [48] with simple appending fusion leads to -0.5% and -1.4% performance drop. This is a strong indication of the difficulty in fusing useful RGB information into the extensive point cloud information. TupleInfoNCE [30] designs a contrastive loss for multimodal representation learning, and contributes to a performance gain of +0.3% on [email protected] from VoteNet baseline with additional RGB and height modalities. In addition to VoteNet, ImVoteNet [47] additionally proposes image votes to boost 3D object detection performance. By plugging our DEQ fusion into ImVoteNet, we obtain +0.8% gain on [email protected] compared to ImVoteNet baseline. Note that the performance of our reproduced ImVoteNet (ImVoteNet repro.) is slightly lower than the one reported in the original paper, and our experiments are based on our reproduced implementation.

VQA-v2. Our experimental results on VQA-v2 based on Mutan [10] and MCAN [67] are shown in Table 5. Mutan [10] initializes GRU with pretrained Skip-thoughts models [25] to process questions, whereas MCAN [67] leverages pretrained GloVe word embeddings [42]. Both methods use bottom-up attention visual features. In addition, MCAN introduces self-attention and guided-attention units to model intra- and inter-modality interactions. Following their basic settings, we replace the fusion method with our DEQ fusion for comparison. We achieve consistent improvements over all evaluation metrics on both baselines, suggesting the superiority of our method.

We conduct extensive ablation experiments to study the effectiveness of our proposed deep equilibrium fusion method from different perspectives. Table 6 details the results. All ablation studies are evaluated on BRCA benchmark using all three modalities, following the same experimental setup stated in Appendix B. Additional ablation studies on other benchmarks are in Appendix C.

Effectiveness of seeking equilibrium. We first examine the effectiveness of computing the equilibrium state to extract and integrate stable modality information at every level. We first discard all components, i.e., directly fusing with a weighted sum approach: $\mathbf{x}_{\mathrm{fuse}}=\sum_{i=1}^{N}w_{i}\mathbf{x}_{i}$, where $w_{i}$ is a learnable weight associated with modality $i$. As shown in Table 6, this baseline fusion method obtains similar performance to [16]. Next, we disable the recursive computation in our DEQ fusion module, i.e., all $f_{\theta_{i}}(\cdot)$ and $f_{\mathrm{fuse}}(\cdot)$ are only applied once without finding the equilibrium states. Since all inputs $\mathbf{z}$ are initialized to zero, this approach is equivalent to the weighted sum approach but with an additional nonlinear projection $f_{\theta_{i}}(\cdot)$ applied to all modality-wise features. Interestingly, introducing additional parameters without DEQ even harms performance compared to the weighted sum baseline. Results from both ablation studies demonstrate the importance of seeking the equilibrium states for multimodal fusion.

Different fusion variants involving DEQ. We compare our DEQ fusion strategy against several variants involving DEQ in Table 6. First, we disable the purified-then-combine fusion strategy, i.e., ablating our fusing projection $f_{\mathrm{fuse}}(\cdot)$ by simply summating all modality-wise features: $\mathbf{z}_{\mathrm{fuse}}^{*}=\sum_{i}^{N}\mathbf{z}_{i}^{*}$. Our full DEQ fusion notably improves all evaluation metrics compared to the runs without the proposed purified-then-combine fusion strategy. Next, we ablate all modality projections $f_{\theta_{i}}(\cdot)$ as identity functions by setting $\mathbf{z}_{i}^{*}=\mathbf{x}_{i}$. Specifically, given a set of features from $N$ modalities $\{\mathbf{x}_{i}\},i=1,2,\dots,N$, we set $\mathbf{z}_{i}^{*}=\mathbf{x}_{i}$. and proceed fusion with $f_{\mathrm{fuse}}(\cdot)$. We notice a decline in all evaluation metrics without modality-wise nonlinear projections. These studies demonstrate that our proposed fusion variant produces the most encouraging results across all evaluation metrics.

Impact of soft gating function. Motivated by the success of dynamically perceiving information from modalities, we develop a soft gating function to capture the important information within each modality. We further validate the effectiveness of the proposed soft gating function $G(\cdot)$. Specifically, we set $\mathbf{z}_{i}^{\prime}=\mathbf{z}_{i}^{[j+1]}$ for Eq. 10 to disable the soft gating function. As shown in Table 7, DEQ fusion without soft gating function leads to about -1% performance drop among all evaluation metrics. Note that since $G(\cdot)$ is a part of $f_{\mathrm{fuse}}$, disabling $f_{\mathrm{fuse}}$ automatically removes $G(\cdot)$. The soft gating function combined with all other components leads to the most superior result.

Convergence of DEQ Fusion. We examine the convergence of our DEQ fusion, which is an important assumption since fusion may collapse if it fails to find the equilibrium. We train a model with our DEQ fusion from scratch, and track the relative difference norm evaluated as ${\|\mathbf{z}_{\mathrm{fuse}}^{[i+1]}-\mathbf{z}_{\mathrm{fuse}}^{[i]}\|}/{\|\mathbf{z}_{\mathrm{fuse}}^{[i]}\|}$ over 100 solver steps during inference. We compare it with a weight-tied fusion approach which simply iterates our fusion layer and performs backward pass layer-by-layer. Fig. 3 depicts the empirical results. It is notable that the difference norm of our DEQ fusion quickly drops below 0.01 on average within 20 solver steps, whereas the weight-tied fusion oscillates around a relatively high value. Benefiting from fixed point solvers and analytical backward pass, our DEQ fusion has much quicker and stabler convergence to the fixed point than the weight-tied approach.