Hierarchical Cross-modal Transformer for RGB-D Salient Object Detection

As handcrafted saliency cues (Ciptadi, Hermans, and Rehg 2013; Peng et al. 2014; Cong et al. 2016) are weak in learning global contexts and hold limited generalization ability, recent RGB-D SOD models mainly focus on designing CNN architectures and cross-modal fusion paths to better study the complements. For example, Qu et al. (2017) integrate handcrafted cues from two modalities as a joint input to train a shared CNN. PCF (Chen and Li 2018) proposes a progressive multi-scale fusion strategy and TANet (Chen and Li 2019) introduces a three-stream architecture to explicitly select complementary cues in each level.

Apart from the basic fusion architectures (i.e., single stream, two-stream, and three-stream), some other works introduce various feature combination strategies (Han et al. 2018), cross-modal cross-level interaction paths (Liu, Zhang, and Han 2020), and other strategies such as knowledge distillation (Chen et al. 2021) and dynamic convolution (Pang et al. 2020) to boost the fusion sufficiency.

In summary, the prior CNN-based RGB-D SOD community has achieved noticable advances. Nonetheless, the locality nature of CNNs make them weak in learning global contexts. Different from previous remediation strategies such as using global pooling (Liu, Han, and Yang 2017) or fully connected layers (Liu and Han 2016a), Liu et al. (2021a) eschew this intrinsic limitation by designing a transformer-based architecture to extract intra/inter-modal long-rang dependencies. They use ViTs as encoders for each modality and stitch the tokens from two modalities for cross-modal fusion. Benefit from the long-range self-attention mechanism, VST achieves large improvement over previous CNN-based methods. Whereas, it ignores the large cross-modal gap. Consequently, directly computing the cross-modal dependency between distant regions contributes little contexts and even tends to introduce noises. Also, simply concatenating features in the decoder without differentiating neglects the varying contributions of different layers in inference. Differently, we customize a hierarchical cross-modal fusion module to carefully model the cross-modal transformer complements from two views, as well as a feature pyramid module to enable adaptive cross-level integration.

The transformer (Vaswani et al. 2017) is first proposed for machine translation and achieves impressive results in various natural language processing tasks with its powerful ability in modeling global contexts. Inspired by its great potential, increasing vision transformers have been put forward in the computer vision community. Dosovitskiy et al. (2020) successfully introduce transformer (ViT) into image classification by splitting an image into patches. Since then, various transformers are introduced, e.g., T2T ViT (Yuan et al. 2021) aggregates adjacent tokens to model local structure and reduce the computational cost, PVT (Wang et al. 2021) proposes spatial-reduction attention to learn multi-scale feature maps flexibly. Swin transformer (Liu et al. 2021b) fully takes advantages of displacement invariance and size invariance and integrates them into an transformer. These transformers are widely applied in other computer vision tasks as backbones, including detection (Carion et al. 2020), segmentation (Kirillov et al. 2019), video processing (Sun et al. 2019), etc.

Transformers are also widely used in multi-modal learning for its flexible input. For instance, Sun et al. (2019) straightly project the video and audio into patches and throws them into a shared transformer backbone. Zheng et al. (2021) design a transformer structure to handle embeded acoustic input and text simultaneously. However, these methods concatenate embeddings straightly and achieve intra-modal and inter-modal combinations at the same time, leading to uninformative fusion and even introducing noisy features. The VST (Liu et al. 2021a) model tackles this issue by extracting RGB and depth features individually and adopting a cross attention module for cross-modal fusion. However, the cross-modal attention module in VST introduces cross-modal gap and spatial discrepancy simultaneously, making it difficult to explore the heterogeneous cross-modal complements clearly. Hence, we decouple the complex cross-modal transformer complements and propose a hierarchical cross-modal attention to control the modality/spatial gaps and progressively incorporate cross-modal complements.

Fig. 2 shows the architecture of our proposed model. Two T2T ViT backbones pretrained on ImageNet are applied to extract RGB and depth features, respectively. And then the HCA module is tailored to explicitly take advantage of the cross-modal spatial alignment to boost the global/local contextual fusion. Next, the feature pyramid is constructed to selectively assimilate shallower layers with the guidance of the deep semantics. Finally, the DCM is introduced to bifurcate the heterogeneous complementary cues to modal-shared and modal-specific ones to boost the fusion sufficiency and their combination is used to generate the final joint prediction.

As shown in Fig. 3 (a), our HCA consists of two stages: Global self-attention (GSA) and Local-aligned cross-attention (LCA) as illustrated in Fig. 3 (b) and Fig. 3 (c), respectively. In the GSA stage, RGB and depth features will calculate their intra-modal self-attention maps individually, and then swap their global self-attention maps to achieve complementary global contexts. The attention scores are calculated by the following formulas:

$$GSA(Q_{r},K_{r},V_{d})=softmax(\frac{Q_{r}K_{r}^{T}}{\sqrt{d}}V_{d}),$$

(1)

$$GSA(Q_{d},K_{d},V_{r})=softmax(\frac{Q_{d}K_{d}^{T}}{\sqrt{d}}V_{r}),$$

(2)

where $Q_{r}$, $K_{r}$, $V_{r}$ denotes query, key and value generated from RGB features, and $Q_{d}$, $K_{d}$, $V_{d}$ come from depth features. Unlike self-attention, we swap the attention map $Q\times K$ which describes the within-modal self-relation globally. The $GSA(Q_{d},K_{d},V_{r})$ and $GSA(Q_{r},K_{r},V_{d})$ will finally add back to RGB and depth features respectively as a residual part.

After swapping the structural information, we adopt a local cross-attention module to reinforce the fusion of cross-modal local semantics. The main difference between LCA and traditional cross attention lies in a mask operation, which will keep attention similarity $Q\times K^{T}$ in adjacent area and set similarity to 0 in remote areas. Specifically, the global cross-attention map $Q\times K^{T}$ will add with a mask matrix $M$ where 0 denotes adjacent area and -100 represents remote area, after softmax the attention scores from remote area are close to 0, thus remote areas make no contribution to the further fusion process. This process can be formulated as follows:

$$LCA(Q_{r},K_{d},V_{d})=softmax(\frac{Q_{r}K_{d}^{T}+M}{\sqrt{d}}V_{d}),$$

(3)

$$LCA(Q_{d},K_{r},V_{r})=softmax(\frac{Q_{d}K_{r}^{T}+M}{\sqrt{d}}V_{r}).$$

(4)

Fig. 4 details the process of mask generation. After embedding RGB and depth features into $Q$, $K$ and $V$ forms, we multiply $Q_{r}$ with $K_{d}^{T}$ to obtain a unified attention map. For example, the first line in attention map depicts the similarity between the first patch in RGB features and all the patches in the paired depth. Then, we design a mask for each line according to its distance to the target patch and get the flitted attention map. The masked attention map, carrying complementary local contexts and cross-modal correlation, is then multiplied with the $V_{d}$ to generate the final depth features. Note that a symmetrical interaction line is also performed to enhance RGB features. To further encourage the extraction and selection of cross-modal complementary cues, updated features will be forced to predict the saliency maps by optimizing:

$$loss1=-[y\cdot\log\sigma(x_{r})+(1-y)\cdot\log(1-\sigma(x_{r}))],$$

(5)

$$loss2=-[y\cdot\log\sigma(x_{d})+(1-y)\cdot\log(1-\sigma(x_{d}))],$$

(6)

where $\sigma$ means sigmoid function, y denotes the groundtruth labels, $x_{r}$ and $x_{d}$ denote the predicted results from RGB feature and depth feature, respectively.

After we have obtained the complemented features from the HCA module, we draw inspiration from the Feature Pyramid Networks (FPN) (Lin et al. 2017) to combine the transformer features from different levels to get multi-level multi-modal representations. Unlike previous methods (Chen, Li, and Su 2019; Liu et al. 2021a; Zhao et al. 2022) that construct feature pyramid using CNN features and distribute the features to each level equally, we highlight the contribution of the deepest level in the pyramid with a larger proportion and use the deep semantics to guide the selection of shallow features.

The architecture of the FPT is shown in Fig. 5. The features (14*14*384, 28*28*64, 56*56*64) from two T2T ViT encoder are aligned by progressively upsampling and concatenated with channel ratios 6:1, 2:1 and 1:1, respectively.

Considering the complementarity among cross-modal features are heterogeneous, we draw inspiration from (Zhao et al. 2022; Chen et al. 2020a) to disentangle the cross-modal complements into consistent and complementary ones to improve the fusion adaptivity. Specifically shown in the DCM in Fig. 6, the input RGB features $F_{A}$ and depth features $F_{B}$ will first be mapped to the same feature space by a linear projection and fed to two branches to extract consistent and complementary features, respectively. The branch above explores the consistency by point multiplication and a residual connection:

$$F_{consistent}=Conv(F_{A}^{map}\otimes F_{B}^{map}\oplus F_{A}^{map}),$$

(7)

where $F_{A}^{map}$ and $F_{B}^{map}$ mean the RGB features and depth features after mapping, $\otimes$ depicts element-wise multiplication, $\oplus$ means element-wise addition and $Conv$ is the convolution layer. The branch below explores the complementarity by substraction:

$$F_{complement}=Conv(\left|F_{A}^{map}\ominus F_{B}^{map}\right|),$$

(8)

where $\ominus$ depicts element-wise subtraction.

To encourage the disentanglement of complements, we append the saliency groundtruth mask to supervise the prediction of the fused feature $F_{fused}$. Specifically, the saliency map $P_{i-1}$, predicted by the former DCM, carries important global contexts and localization cues. Hence, we use $P_{i-1}$ to guide this disentanglement in the i-th level by point multiplication with $F_{consistent}$ and $F_{complement}$ respectively to avoid noisy backgrounds in shallower layers. The constrained features are calculated as following formulas:

$$F_{consistent}^{\prime}=Conv(F_{consistent}\otimes P_{i-1}),$$

(9)

$$F_{complement}^{\prime}=Conv(F_{complement}\otimes P_{i-1}).$$

(10)

Finally, $F_{consistent}^{\prime}$ and $F_{complement}^{\prime}$ will be combined with an addition and a convolution operation:

$$F_{fused}=Conv(F_{consistent}^{\prime}\oplus F_{complement}^{\prime}).$$

(11)

The prediction loss in the i-th DCM is

$$loss_{i}=-[y\cdot\log\sigma(P_{i})+(1-y)\cdot\log(1-\sigma(P_{i}))],$$

(12)

where $P_{i}$ is the saliency map generated by the i-th DCM.

Compared with undifferentiated concatenation, our proposed DCM will explicitly diversify the complex cross-modal complements into the consistent and complementary elements.

The final loss includes 2 losses from the HCA module and 4 losses from DCMs:

$$loss_{final}=loss_{r}+loss_{d}+\sum_{i=1}^{4}loss_{i}.$$

(13)

We evaluate the proposed model on seven public RGB-D SOD datasets which are NJUD (Ju et al. 2014) (1985 image pairs), NLPR (Peng et al. 2014) (1000 image pairs), DUTLF (Piao et al. 2019)(1200 image pairs), STERE (Niu et al. 2012) (1000 image pairs), RGBD135 (Cheng et al. 2014) (135 image pairs), SIP (Fan et al. 2020a) (929 image pairs) and COME15K (Zhang et al. 2021) (15625 image pairs). We follow the consistent setting in previous works (Liu et al. 2021a; Zhang et al. 2021) and choose 1485 image pairs in NJUD, 700 image pairs in NLPR, 800 image pairs in DUTLF and 8025 pairs image in COME15K as the training set and the remaining are for testing. Similarly. we also adopt some data augmentation techniques such as resize, random crop and random flipping to avoid overfitting.

We adopt four widely used evaluation metrics to evaluate our model. Specifically, Structure-measure $S_{m}$ (Fan et al. 2017) evaluates region-aware and object-aware structural similarity. E-measure $E^{max}_{\rho}$ (Fan et al. 2018) simultaneously considers pixel-level errors and image-level errors. Maximum F-measure (Achanta et al. 2009) jointly considers precision and recall under the optimal threshold. Mean Absolute Error (MAE) computes pixel-wise average absolute error.

We implement our model on the base of VST (Liu et al. 2021a) using Pytorch and train it on a RTX 3090. The training parameters are set as follows: batch size is 8, epoch is 50. For the optimizer, we use Adam with the learning rate gradually decaying from $10^{-4}$ to $10^{-6}$.

To quantitatively measure our model, we compare it with 6 SOTA RGB-D SOD methods, including CMW (Li et al. 2020), Cas-Gnn (Luo et al. 2020), HDFNet (Pang et al. 2020), CoNet (Ji et al. 2020), BBS-Net (Fan et al. 2020b) and VST (Liu et al. 2021a). Tab. 1 shows the comparison in terms of the S-measure, F-measure, E-measure and MAE scores. To fairly demonstrate the advantages of our cross-modal fusion scheme, we select the VST without the edge detection task and retrain it with our training set as the baseline model (denoted by ”base”), as we argue that edge detection is a trick requiring additional edge labels and not related to multimodal fusion. The quantitative results illustrate that VST achieves consistent improvement over CNN-based methods, denoting the superiority of the transformer. Our model outperforms previous RGB-D SOD models, including VST on all datasets, demonstrating the advantages of our cross-modal fusion scheme and designs.

To visually measure our model, we compare it with 3 recent SOTA RGB-D SOD methods, including CoNet (Ji et al. 2020), BBS-Net (Fan et al. 2020b) and the Base(Liu et al. 2021a). The visualized results on some representative challenging scenes are illustrated in Fig. 7. In the case of large intra-difference in the foreground (see the $1^{st}$, $2^{nd}$ and $4^{th}$ rows), previous models often fail to completely detect the correct salient objects, while our model can achieve more accurate and uniform detection. When the foreground and background hold similar appearance or depth (see the $1^{st}$, $3^{rd}$ and $6^{th}$ rows), previous methods may mistake some background areas as foreground, while our model successfully handles this confusion and well leverage the discriminating modality. For scenes having multiple salient objects (see the $2^{nd}$, $5^{th}$ and $6^{th}$ rows), other models tends to overlook some regions, while our method can highlight all salient objects. The success on these difficult cases demonstrates our advantages in modeling within/cross-modal long-range dependencies and local correlations.

In this section, we will verify the effectiveness of each proposed structure by comprehensive ablation experiments. The experiments are implemented on 2 largest datasets, i.e., COME-EASY and COME-HARD. We select the Base as the baseline model (noted as ”Base”).

Effectiveness of hierarchical cross-modal attention. As shown in Tab. 2, compared to the original VST baseline (denoted by ”Base”), removing the global cross-attention layer in VST (denoted by ”Base-CA”) surprisingly improves the performance, suggesting the cross-modal gap and the irrationality of using spatial-distant cross-modal dependencies as complementary contexts. In contrast, our hierarchical cross-modal attention scheme (denoted by ”+HCA”) shows noticeable improvement, which well supports our motivation that cross-modal gap and spatial discrepancy should not be concurrent when measuring cross-modal correlations.

Fig. 8 shows the effectiveness of the global cross-attention (GSA) and local-aligned cross attention (LCA) components in HCA visually. As we predict, the global cross-attention in VST (see ”Base” in Fig. 8) will introduce noises from other distant areas to wrongly identify the background and foreground. With global self-attention (denoted by ”+GSA”) to complement the global contexts, the model has stronger global reasoning ability to localize the salient object. With the local-aligned cross attention additionally (denoted by ”+HCA”), the cross-modal fusion process takes advantages of the local cross-modal correlations to remove the noises and enhance the object details.

Effectiveness of feature pyramid for transformer. The comparison between ”+HCA+FPT” and ”+HCA” in Tab. 2 illustrates the improvement of our FPT design, indicating its contribution to boost the cross-level feature selection. The saliency maps in Fig. 9 visually shows the benefits of FPT. In FPT, cross-level features are integrated progressively and selectively with the guidance from deep to shallow, thus enabling informative and adaptive multi-level combination. As a result, our model can better infer the location of the salient objects and highlight their boundaries precisely.

Effectiveness of the disentangled complementing module. The comparison between line ”+HCA+FPT” and ”+HCA+FPT+DCM” in Tab. 2 shows the considerable improvement of DCM, indicating that diversifying the complex cross-modal complements into consistent and complementary ones will boost the fusion adaptivity.

The visualization results are shown in Fig. 10. Undifferentiated concatenation is too ambiguous to explore the detailed consistent information between modalities, thus the predicted maps are bad in details, especially around the object boundaries (see the $1^{st}$, $3^{rd}$ and $4^{th}$ rows). Instead, DCM explicitly decouples the difference and consistency between modalities and combine them adaptively for varying scenes to localize the salient objects accurately and highlight salient regions uniformly.