Part-Whole Relational Fusion Towards Multi-Modal Scene Understanding

Part-level modality capsules disentanglement targets at disentangling the full-resolution capsule features of each single modality to two orthogonal versions, including horizontal and vertical capsules, which will transform two-dimensional capsule routing to one-dimensional version with computation reduced greatly.

Given features $\left\{{\bf{f}}_{i}^{n}\in{\Re^{H_{i}\times W_{i}\times C_{i}}}\mid n\in\left\{1,2,3\right\},i=1,2,3,4\right\}$ from the backbone network of ResNet-50 [48], where $H_{i}\times W_{i}$ and $C_{i}$ represent the resolution and channel of features at stage-$i$ for modality $n$, respectively. Each single modality capsule, including pose matrix ${\bf{p}}_{i}^{n}$ and activation value $\boldsymbol{a}_{i}^{n}$, can be constructed as follows:

where $Conv\left({*,dim=3}\right)$ and $\sigma$ refer to convolution and Sigmoid operations along the 3${rd}$ channel, respectively. $T^{p}$ is the type number of the part-level capsules. 16 and 1 represents the pose matrix dimension and activation value dimension, respectively. The full-resolution capsules ${\mathbf{P}_{i}^{n}}$ is composed by

On top of the full-resolution capsules ${\mathbf{P}_{i}^{n}}$, we disentangle the horizontal 1-dimensional capsules as follows

Similarly, the vertical part-level capsules can be disentangled as

The disentangled horizontal and vertical part-level modal capsules, ${\mathbf{P}_{i,H}^{n}}$ and ${\mathbf{P}_{i,V}^{n}}$, will be fed into the capsule routing algorithm [15] to find the whole-level modal capsules via exploring part-whole relations.

Specifically, part-level capsules of multiple modalities are combined together, which is achieved by concatenating horizontally part-level capsules of multiple modalities and vertically part-level capsules of multiple modalities along the type dimension separately, i.e.,

The horizontally and vertically whole-level modal capsules will be computed by implementing capsule routing [15] on horizontally and vertically part-level modal capsules separately as follows

where $Routing\left(*\right)$ represents the capsule routing algorithm [15]. $\mathbf{R}_{i,H}$ and $\mathbf{R}_{i,V}$ are the routing coefficients from part-level modalities to the whole-level modal, which reveal the familiar relations between the part-level modalities and the whole-level modality along the horizontal and vertical dimensions, respectively. ${\mathbf{W}_{i,H}}$ and ${\mathbf{W}_{i,H}}$ are the horizontal and vertical whole-level modalities, respectively.

On top of that, the full-resolution whole-level modal capsules can be achieved by entangling them as follows

where $\otimes$ represents the operation of matrix multiplication along the resolution dimension.

The whole-level modality ${{\bf{WP}}_{i}}$ in Eq. (9) captures associations across different individual modalities, since it is computed by routing information from multiple modalities. The generation of modal-shared details is a crucial step in the PWRF framework, allowing the model to leverage information that is consistent across multiple modalities. The modal-shared details are generated by aggregating features from all part-level modalities through a part-whole relational routing mechanism. In light of this fact, we treat it as the modal-shared semantic details.

The routing coefficients $\mathbf{R}_{i,H}$ and $\mathbf{R}_{i,V}$ in Eqs. (7) and (8) indicate the likelihood of each part-level capsule belonging to each latent whole-level capsule class, which reveal the contributions of different single part-level modalities to the whole-level modality, respectively. Therefore, we model the modal-shared details for each modality using the routing coefficients and each modalities capsules. First, the part-level correlations for each modality can be extracted from the horizontal and vertical routing coefficients $\text{R}_{i,H}$ and $\text{R}_{i,V}$

where $\text{R}_{i,H}^{n}$ and $\text{R}_{i,V}^{n}$ represent the horizontal and vertical routing coefficients for each part of the capsule corresponding to $n^{\text{th}}$ modality, respectively. The third dimension of the routing coefficients is split into three parts, each of size $T^{p}$.

The horizontal and vertical modal-specific details can be computed via multiplying the each part-level correlations and its capsule features

where ${{\odot}}$ means element-wise multiplication. The full-resolution modal-specific details are further computed by entangling ${\bf{SP}}_{i,H}^{n}$ and ${\bf{SP}}_{i,V}^{n}$ along the resolution dimension as follows

It is noted that each modality specific details ${\bf{SP}}_{i}^{n}$ is reshaped to ${\bf{SP}}_{i}^{n}\in{\Re^{{H_{i}}\times{W_{i}}\times({T^{p}}\times 17)}}$ by merging the last two dimensions together for the following utilization. On top of that, modal-specific components ${\bf{SP}}_{i}^{n}$ of multiple modalities are integrated to get the merged modal-specific details as

where $ConvCate(*)$ denotes the operation that combines concatenation and convolution operation along the specified dimension.

To make the entire routing process more intuitive, Fig. 2 visualizes the the process of spliting the routing coefficients. As shown in Fig. 2, on top of the routing between the high-level capsules and the low-level capsules, the routing coefficients are obtained. In order to obtain the corresponding routing coefficients for the respective modalities, the routing coefficients are split and averaged along the last dimension. The reshape operation is gone through for subsequent processing.

In view of the complementary characteristic of modal-shared and modal-specific cues with respect to the input, their primitive integration will benefit the semantic exploration. To this end, we design a shared-specific integration module to combine complementaries of modal-shared and modal-specific details for better semantic inference, which consists of two components, including primitive modal-specific details generation and shared-specific interaction.

A. Primitive modal-specific details generation

In order to attenuate noise of each modal-specific information, primitive modal-specific information is generated by using the original modality features ${\bf{f}}_{i}^{n}$ and modal-shared details $\mathbf{F}_{i,{shd}}$ as follows

where the modal-shared details, denoted as $\mathbf{F}_{i,{shd}}$, are derived from reshaping the whole-level modal capsules $\mathbf{WP}_{i}$. The primitive modal-shared details can be derived as

As such, a more primitive modal-specific information is achieved by combining three single modalities together as follows

B. Shared-specific interaction

To integrate modal-shared and modal-specific details, we propose a shared-specific interaction module. Besides modal-shared details $\mathbf{F}_{i,{shd}}$ and primitive modal-specific details $\mathbf{F}_{i,{psg}}$, we also employ the selected modal details of Self-Query Hub[14] denoted as $\mathbf{F}_{i,{sqh}}$. Specifically, three parallel branches are first designed to interact these three components. Within each branch, one component is selected as the primary cue, while the remaining two components are utilized to attend the primary cue for better semantic exploration, which is achieved by a spatial attention and a channel attention. The spatial attention is implemented as follows

where ${\bf{CP}}^{1}_{i}$ is the primary component. ${\bf{CP}}^{2}_{i}$ and ${\bf{CP}}^{3}_{i}$ are the remaining two components. $CGMP(*)$ denotes the global max pooling operation performed along the channel direction.

The channel attention is implemented as

where $GMP(*)$ refers to the adaptive global max pooling operation.

Based on the spatial and channel attentions, the primary component can be attended as

Doing Eqs. (20)-(22), we obtain the shared-specific interaction for each primary component ${\bf{CP}}_{i}^{j},j=1,2,3$.

The interacted component ${\bf{F}}_{i,ssi}$ in three branches are integrated to get the merged information using the element-wise multiplication and element-wise addition, which is written as

where $\oplus$ represents the element-wise addition.

On top of the fusion features ${\bf{F}}_{i}^{u}$, we integrate it with the primary RGB modality using the fusion step [14] to get the multi-modal fusion features, which is fed into the multi-layer perception decoder [49] to predict the semantic result ${\bf{Pre}}$. To train the model, the online hard example mining cross-entropy loss function [50] is used to compute the difference between the semantic predictions and the ground truth ${\bf{GT}}$, i.e.,

High-level features contain rich semantic information, encapsulating the overall properties of salient objects. In contrast, low-level features preserve edge details of salient objects. To complement their superiority, we design an ASA module to integrate adjacently high-level and low-level semantics for modal-shared and modal-specific in Fig. 5, which is composed by three components, including adjacent-scale integration, dual-branch attention, and selective aggregation.

Adjacent-scale integration. The adjacent-level modal-shared details (${\bf{WP}}_{i-1}$ and ${\bf{WP}}_{i}$) and modal-specific details (${{\bf{SP}}_{i-1}}$ and ${{\bf{SP}}_{i}}$) are integrated via

${\bf{WP}}_{i}^{\prime}$ and ${\bf{SP}}_{i}^{\prime}$ can be obtained by

where $CBR(\cdot)$ and $Bilinear$ represent the operations of (Convolution + BatchNorm + ReLU) and bilinear upsampling, respectively.

Dual-branch attention. A dual-branch attention including local attention and global attention is designed to attend the informative regions. Specifically, the dual-branch attention is achieved by

where $CBRCB(\cdot)$ is the local attention using (Convolution + BatchNorm + ReLU + Convolution + BatchNorm). $ACRC(\cdot)$ is the global attention using (Average pooling + Convolution + ReLU + Convolution).

Selective aggregation. To address the feature discrepancy, a selective aggregation strategy is designed to suppress redundant information and prevent feature contamination. To this end, a gate signal mechanism is introduced to aggregate adjacent-level modal-shared and modal-specific details, i.e.,

As illustrated in Fig. 4(b), we stack two sub-decoders composed by ASA to improve the feature aggregation to produce saliency maps, which implement features aggregation following bottom-up and top-down flows. Specifically in the bottom-up process, the ASA within each decoder progressively aggregates from high-level to low-level features for both modal-shared and modal-specific, separately. The resulting aggregated features ${{\bf{F}}^{asa}_{i,shd}}$ and ${{\bf{F}}^{asa}_{i,spc}}$ contribute to the generation of a preliminary saliency map. Conversely, in the top-down process, the shallowest aggregated features of the first sub-decoder are densely used to guide the second sub-decoder to learn primitive features. By the way, edge cues are employed to enhance the depth feature maps by adjusting their poor conditions. This process improves boundary refinement, allowing for more accurate delineation of object edges and a better representation of spatial structures within the depth maps. Aggregated features of two sub-decoders are combined to generate the final saliency prediction.

The loss function $L$ is defined as:

where $L_{B}$, $L_{S}$, and $L_{I}$ represent binary cross entropy loss [52], structural similarity loss [53], and intersection-over-union loss [54], respectively. $\mathbf{Pre}_{i}$ represent the $ith$ predicted saliency map. $\mathbf{GT}$ is the Ground Truth.

Table 1 provides a comprehensive comparison between our approach and some State-Of-The-Art (SOTA) methods, including HRFuser [13], SegFormer [49], TokenFusion [11], CMX [28], and CMNeXt [14], on the DELIVER dataset [14]. Following [14], we list evaluation values of different models on the validation set of DELIVER [14] in Table 1. In Table 1, most cross-modal fusion approaches are beaten by the multi-modal fusion methods. Besides, our model achieves the best IoU values over the multi-modal fusion methods, which proves the superiority of our model over the other methods. To be more concrete, the IoU value for each class are listed in Table 2. For more comprehensive comparison, we test our model and the SOTA multi-modal fusion method CMNeXt [14] on the test set of the DELIVER dataset [14]. The mIoU values of our model and CMNeXt [14] are 54.29% and 53%, respectively, which proves a significant improvement of our model over the SOTA method.

For more concrete analysis, we conduct a comparison against established multi-modal fusion methodologies across varying conditions, including adverse weather and partial sensor failure scenarios. As shown in Table 3, single-modal and cross-modal fusion methods exhibits limitations in adverse scenarios due to less auxiliary modalities. Although multi-modal fusion methods, e.g., HRFuser [13] and CMNeXt [14], obtain good performance, our model is superior on most adverse scenarios thanks to the primitive fusion for multiple modalities.

To evaluate the effectiveness of our PWRF for VDT salient object detection, we compare it with 13 state-of-the-art approaches, including two RGB salient object detection methods (CPD [59] and RAS [60]), three RGB-D salient object detection methods (BBSNet [61], DPANet [62], and RD3D [63]), six RGB-T salient object detection methods (CGFNet [65], CSRNet [53], DCNet [66], and LSNet [67], SwinNet [55], and HRTransNet [55]), and two VDT salient object detection methods (HWSI [38] and MFFNet [12]). To ensure fair comparisons, all model predictions are either provided by the authors or generated using their source codes with default settings.

As shown in Table 4, three findings can be easily concluded: i) Compared with the RGB methods, most cross-modal methods achieve better performance, which is owing to the complementaries of depth and thermal modalities; ii) Compared with the V-D and V-T approaches, VDT methods both get higher metric values, which thanks to the complementary of triple modalities; iii) Compared with the previous VDT methods, our method achieves consistently superior performance for five best evaluation metrics, which owes to the superior multi-modal fusion of PWRF over the previous simple fusion mechanisms. To be more concrete, compared to the second-best model, MFFNet [12], our model achieves significant improvements. Besides, our framework on cross-modal settings, including VD and VT, consistently achieves superior performance, which can be found from the second and third brackets in Table. 4, where our model on VD and VT conditions outperforms better than the other methods.

In addition, to demonstrate the robustness of our method in handling challenging scenarios, we present the performance comparison for visible-challenge, depth-challenge, and thermal-challenge scenes in Table 5, Table 6, and Table 7, respectively. For the challenging scenarios, our method still achieves a superior performance.

To visually demonstrate the performance of different models for SMM semantic segmentation, we selected two representative scenarios for comparison: “Cloud & Underexposure" and “Rainy & LiDAR Jitter". As shown in Fig. 7²²2The values of semantic labels are scaled via $\times 10$ for visualization in Fig. 7., compared with the single-modal method SegFormer [49], multi-modal fusion methods provide more accurate semantic analysis. Moreover, compared with the multi-modal fusion method CMNeXt [14], our method benefits from the Part-Whole Relational Fusion (PWRF) framework, resulting in more complete object segmentation in complex scenarios.

To visually demonstrate the superiority of our model, we present several visualization results from challenging scenes across visible, depth, and thermal images. Specifically, V-challenges include big salient object (BSO), low illumination (LI), multiple salient objects (MSO), no illumination (NI), similar appearance (SA), side illumination (SI), and small salient object (SSO). D-challenges contain background interference (BI), background messy (BM), incomplete information (II), and small salient object (SSO). T-challenges cover crossover (Cr), heat reflection (HR), and radiation dispersion (RD). As shown in Fig. 8(a), (b), and (c), the proposed model well tackles these challenging scenes for salient object detection, compared with the other methods.

In this subsection, we will conduct ablation experiments on DELIVER [14] to evaluate the contributions of key components of the SMM semantic segmentation model.

Different number of capsule types for PWRF. Capsule type number takes a vital role in part-whole relations exploration, which will affect the performance of the whole model. To take a thorough study on different number of capsule types, we run several rounds with different part-level capsule types³³3The whole-level capsule types number is set to the category number for different datasets.. As shown in Table 8, it is seen that few or more capsules will lower the performance, because few capsules cannot find the accurate part-whole relations while more capsules will introduce noisy capsule assignments. By contrast, 8 types achieve the best IoU value, which is the setting for our model in this paper.

Shared parameters. Since there are multiple stages, our PWRF architecture should be repeated multiple times for different modality branch, which generates multi-branch structures. To discuss parameters sharing for these consistent structures, we carry out experiments using shared and unshared parameters for the model. As listed in the last two columns of Table 8, shared parameters improve the semantic segmentation performance compared with the unshared setting. The reason behind comes from three folds: i) Shared parameters reduce some noise caused by the modality gap; ii) Shared parameters learn consistent fusion trend for different modalities; iii) By sharing structures and parameters, data from different modalities can assist each other to enhance the understanding of the same scenario.

In this subsection, we conduct ablation experiments on VDT-2048 dataset to evaluate the contributions of key modules in our proposed method.

Different components. To verify the effectiveness of different components in our proposed method, we perform various ablation experiments in Table 9. First, comparing (a) & (b) and (a) & (c) in Table 9, the proposed PWRF and stacking ASA decoder significantly boost the performance. The performance improvements come from two aspects: i) PWRF dynamically captures the informative semantics from different modalities for fusion; ii) Stacking ASA decoder helps to extract the primitive context of different modalities for prediction. Secondly, comparing (b) & (d) and (c) & (d) in Table 9, the combination of PWRF and stacking ASA decoder achieves a higher performance, which proves the contributions of the proposed components efficiently.

Different fusion mechanisms. In order to investigate the contribution of our PWRF for triple-modal fusion, we carry out several experiments by replacing our PWRF with different fusion mechanisms, including addition, concatenation, QKV attention mechanisms and EM routing [15] in our VDT salient object detection model. As shown in Table 10, there are two findings: i) Our PWRF surpasses the simple addition and concatenation mechanisms due to the primitive fusion for multiple modalities; ii) Compared with the attention mechanism, our model still achieves a significant superiority; iii) Our PWRF outperforms the previous EM routing [15] with a large margin, which demonstrates the superiority of DCR routing in our PWRF for VDT salient object detection.

Stacking ASA decoder. Stacking ASA decoder contains two sub-decoders using a bridge connection. To deeply dig into its contribution, we conduct experiments including baseline by removing stacking ASA decoder from the entire model, one sub-decoder, and two sub-decoders. As shown in Table 11, compared with baseline, one sub-decoder definitely improves the performance, which is because ASA emphasizes semantic feature channels while suppressing noisy ones. In addition, compared with one sub-decoder, stacking two sub-decoders performs better, which proves the bridge connection of two sub-decoders helps the model to get the superior performance.

Ablation analysis on different modalities. To assess the impact of different modalities, we conduct four experiments detailed in Table 12, focusing on modalities combinations such as V+D, V+T, D+T, and V+D+T. Our PWRF method necessitates utilizing at least two distinct features from each modality, hence experiments on individual modalities were omitted. Moreover, for experiments involving combinations like V+D, V+T, and D+T, we simply removed one capsule feature branch while keeping other operations unchanged. From Table 12, it is evident that V+T obtains good performance compared with V+D and D+T. Leveraging three modalities prefer to improve the performance significantly, which demonstrates the superiority of more-modal fusion over cross-modal fusion.

Most previous methods cannot interpret the fusion of different modalities, which limits their reliability in real-world applications. In contrast, our model can provide an explanation for multi-modal fusion due to the part-whole relational routing from part-level modalities to whole-level modal. Specifically, we plot the horizontal and vertical routing coefficients for one pixel in Fig. 9, in which higher routing values represent higher contribution of single modalities for fusion. In Fig. 9, we observe differences in the contributions from each modality under horizontal and vertical routing conditions. The x-axis, y-axis, and z-axis represent the part-level capsule types, whole-level capsule types, and routing coefficients, respectively. For example, as shown in the red rectangle of Fig. 9(a) in terms of the horizontal dimension, from the 8th part-level capsule to the first whole-level capsule, depth and event modalities contribute much for fusion, while LiDAR contributing less, which explains that depth and event modalities occupies much for semantic understanding while LiDAR has no roles at this pixel. By contrast, in the vertical direction at the pixel, other modalities might dominate. Such differences could be due to the unique characteristics captured by each modality in different spatial dimensions. These directional differences in contribution help capture complementary information from different modalities, thus enhancing the feature fusion.