Robust Multimodal 3D Object Detection
via Modality-Agnostic Decoding and Proximity-based Modality Ensemble

Cross modal transformer (CMT) [36] is a recent framework that uses a transformer decoder to aggregate information from both modalities into object queries. In CMT, the modality-specific backbone first extracts modality features e.g., VoVNet [8] for camera and VoxelNet [45] for LiDAR. Then, they localize 3D bounding boxes using a transformer decoder $f$ with modality-specific position embeddings that help object queries aggregate information from both modalities simultaneously. Given the flattened LiDAR BEV features $X_{L}\in\mathbb{R}^{(H_{L}*W_{L})\times D}$ and camera image features $X_{C}\in\mathbb{R}^{(N_{C}*H_{C}*W_{C})\times D}$, CMT can be formulated as:

$$Z=f(Q,[X_{L};X_{C}]),$$

(1)

where $Q$ denotes a set of learnable object queries and [;] indicates the concatenation. $H_{L},W_{L},H_{C},$ and $W_{C}$ denote the height and width of the LiDAR BEV feature map and camera image feature map respectively, and $N_{C}$ indicates the number of cameras.

CMT is an effective framework but it has some drawbacks discussed in Sec. 1. First, when a LiDAR is missing, CMT lacks the ability to extract geometric information from the camera modality, resulting in substantial performance degradation. Second, when a specific modality shows corrupted signals, information from it may act as noise when aggregating information from both modalities simultaneously. In the following sections, we will demonstrate how to mitigate these issues.

For the cross-modal transformer to maximize the use of both modalities, both geometric and semantic information should be extracted from each modality without relying on a specific one. To this end, we propose a modality-agnostic decoding (MOAD) training scheme, that allows a transformer decoder to fully decode information for 3D object detection regardless of input modalities.

First, we randomly sample learnable anchor positions in the 3D space from uniform distributions and set their positional embeddings to the initial object queries $Q\in\mathbb{R}^{N\times D}$ following PETR [14]. Then $Q$ is processed by multiple decoding branches, each using different modality combinations as an input. This can be formulated as:

	$\displaystyle Z_{LC}=f(Q,[X_{L};X_{C}]),$		(2)
	$\displaystyle Z_{L}=f(Q,X_{L}),$		(3)
	$\displaystyle Z_{C}=f(Q,X_{C}),$		(4)

where $f$ is a shared transformer decoder and $Z_{LC},Z_{L}$, and $Z_{C}$ denote box features from each modality decoding branch. Note that $Q$ is shared across multiple decoding branches. We generate modality-specific positional embeddings $\Gamma_{L}$ and $\Gamma_{C}$ following [14, 36], and $\Gamma_{L}+\Gamma_{C},\Gamma_{L}$, and $\Gamma_{C}$ are used as positional embedings for queries when generating $Z_{LC},Z_{L}$, and $Z_{C}$ respectively. Then, we predict the final box prediction $\hat{B}_{m}$ and classification score $\hat{P}_{m}$ via modality-agnostic box head $h$:

$\displaystyle\hat{B}_{m}=h^{\text{reg}}(Z_{m}),\ \hat{P}_{m}=h^{\text{cls}}(Z_{m}),$

(5)

where $m=\{LC,L,C\}$ indicates the set of modality decoding branch.

We use Hungarian matching between ground truth boxes and predicted boxes from each modality decoding branch respectively for loss computation. The loss function from each modality decoding branch $\mathcal{L}_{m}$ can be formulated as:

$\displaystyle\mathcal{L}_{m}=\omega_{\text{reg}}\mathcal{L}_{\text{reg}}(B,\hat{B}_{m})+\omega_{\text{cls}}\mathcal{L}_{\text{cls}}(P,\hat{P}_{m}),$

(6)

and overall loss function of MOAD is defined as:

$\displaystyle\mathcal{L}_{\text{MOAD}}=\omega_{LC}\mathcal{L}_{LC}+\omega_{L}\mathcal{L}_{L}+\omega_{C}\mathcal{L}_{C}.$

(7)

We use the focal loss for classification loss and L1 loss for box regression.

Note that applying Hungarian matching and computing loss function separately for each modality decoding branch helps fully decode information regardless of input modality. By minimizing the shared transformer decoder with the loss function $\mathcal{L}_{L}$ and $\mathcal{L}_{C}$, modality backbones and transformer decoder learn to extract both geometric and semantic information from each modality without relying o3n specific modality. In addition, minimizing $\mathcal{L}_{LC}$ helps the shared transformer learn how to fuse rich information from both modalities effectively. At test time, we only use the multi-modal decoding branch for final box prediction, resulting in higher performance without additional computational cost.

As discussed in [30, 19], there are challenging environments where one modality outperforms the other, such as at night or sensor malfunction environments. In this case, aggregating information from both LiDAR and camera modalities simultaneously may suffer from negative fusion, which leads to poor box prediction as discussed in Sec. 1. To this end, we propose a novel proximity-based modality ensemble (PME) module that adaptively integrates box features from every modality decoding branch of MOAD.

Given the set of the box features $Z_{m}$ from each modality decoding branch, $Z_{LC}$ is updated through a cross-attention layer with the whole output features including itself. To avoid the noise transfer caused by the interaction between irrelevant box features, we introduce an attention bias $M$ for the cross-attention. To obtain the attention bias, we measure the center distance between the predicted boxes from $Z_{LC}$ and those from $Z_{LC},Z_{L}$, and $Z_{C}$. Then, we apply a linear transformation with a learnable scaler $\alpha\in\mathbb{R}^{1}$ and a bias $\beta\in\mathbb{R}^{1}$. Given $Z_{m}$ and their corresponding center $x,y$ coordinates $\hat{\Phi}_{m}=\{\hat{\phi}_{m,1},\ldots,\hat{\phi}_{m,N}\}$ in the BEV space, the attention bias $M\in\mathbb{R}^{N\times 3N}$ can be formulated as:

	$\displaystyle\hat{\Phi}_{A}=[\hat{\Phi}_{LC};\hat{\Phi}_{L};\hat{\Phi}_{C}],$		(8)
	$\displaystyle M_{i,j}=\alpha*||\hat{\phi}_{LC,i}-\hat{\phi}_{A,j}||+\beta.$		(9)

Then, we add attention bias $M$ to the attention logit and apply the softmax function to calculate attention scores.

Every set of the box features $Z_{LC},Z_{L}$, and $Z_{C}$ are linearly projected by modality-specific projection function $g_{LC},g_{L}$, and $g_{C}$ before the cross-attention layer. To summarize, the overall architecture of the PME can be written as:

	$\displaystyle Q_{m}^{\prime}=g_{m}(Z_{m}),\ \forall m\in\{LC,L,C\},$		(10)
	$\displaystyle Z_{e}=f_{e}(Q_{LC}^{\prime},[Q_{LC}^{\prime};Q_{L}^{\prime};Q_{C}^{\prime}],M),$		(11)

where $f_{e}$ is a cross-attention layer for the modality ensemble. We generate positional embeddings for queries and keys using $\hat{\Phi}_{LC},\hat{\Phi}_{L}$, and $\hat{\Phi}_{C}$ and MLPs. The final box prediction $\hat{B}_{e}$ and $\hat{P}_{e}$ are generated by the box head $h_{e}$ which is another box head for the ensembled box features $Z_{e}$, as:

$\displaystyle\hat{B}_{e}=h_{e}^{\text{reg}}(Z_{e}),\ \hat{P}_{e}=h_{e}^{\text{cls}}(Z_{e}).$

(12)

For proximity-based modality ensemble loss, we apply Hungarian matching between ground truth boxes and predicted boxes. The loss function of PME is defined as:

$\displaystyle\mathcal{L}_{\text{PME}}=\omega_{\text{reg}}\mathcal{L}_{\text{reg}}(B,\hat{B}_{e})+\omega_{\text{cls}}\mathcal{L}_{\text{cls}}(P,\hat{P}_{e}),$

(13)

where $\mathcal{L}_{\text{reg}}$ and $\mathcal{L}_{\text{cls}}$ is the L1 loss and the focal loss respectively following Eq. 6.

Note that the cross-attention mechanism helps our framework adaptively select promising modalities depending on the environment when predicting the final box. MEFormer with PME shows remarkable performance in challenging environments, which will be discussed in Sec. 4.4.

We compare our proposed model with existing baseline models on the nuScenes dataset. To evaluate the model on the test set, we submitted the detection results of MEFormer to the nuScenes test server and report the performance. Note that we did not use any test-time augmentation or ensemble strategies during the inference. As shown in Table 1, MEFormer outperforms other baseline models in terms of both mAP and NDS with a significant margin. Specifically, MEFormer attains a remarkable 73.9% NDS and 71.5% mAP on the nuScenes validation set. On the test set, our model achieves the best performance with 72.2% mAP and the second best performance with 74.3% NDS compared to the previous methods. This demonstrates that, although MEFormer is originally proposed for tackling the sensor missing or corruption scenarios, our proposed approach is effective for enhancing overall detection performance as well.

We present ablation studies of the proposed training strategy and module in Table 5. All experiments are conducted on nuScenes validation set. First of all, as shown in (b), adapting PME improves detection performance by 0.2% NDS and mAP compared to (a) which is identical to CMT. Note that inputs $Z_{L}$ and $Z_{C}$ for the PME in (b) are generated by a transformer decoder trained with only the multi-modal decoding branch since our modality-agnostic decoding is not yet applied. This shows that PME alone helps improve the detection performance. Next, (c) shows applying modality-agnostic decoding enhances the performance by 0.8% NDS and 1.0% mAP. Note that only a multi-modal decoding branch is used for the box prediction in (c) during inference time and there is no additional computation compared to (a). This empirical evidence verifies that reducing reliance on a LiDAR sensor improves the detection performance in overall environments. In (d), we extend our methodology to include both modality-agnostic decoding and proximity-based modality ensemble. Applying both MOAD and PME shows 1.0% NDS and 1.2% mAP performance gain compared to (a), resulting in state-of-the-art detection performance. Performance comparison between (c) and (d) in challenging environments is discussed in Section 4.4.2 and shows a larger performance gap compared to that in the overall environments, which means PME is more effective as the environment becomes more challenging.

We compare the inference speed of our framework with previous frameworks and the result is shown in Table 6.

MEFormer shows 1.0% NDS and 1.2% mAP performance improvement with only a 0.3 FPS reduction in inference speed. Note that all results are measured on a single NVIDIA RTX 3090 GPU and voxelization time is included following [36].

In this section, we validate the effectiveness of our framework with qualitative results. First, Figure 4 shows that the transformer decoder trained with MOAD utilizes the camera to successfully localize the truck that LiDAR fails to detect, resulting in bounding boxes in the multi-modal decoding branch as well. However, without MOAD, all decoding branches fail to detect the same truck. This result validates that MOAD reduces the LiDAR reliance problem in the modality fusion process and helps the framework utilize geometric information in camera modality.

Figure 5 shows qualitative results of MEFormer in the dark environment, where the cameras show extremely corrupted signals. In the front right camera, our framework successfully detects two cars that are hard to recognize with the camera. Additionally, in the back view camera, our framework also detected the car in which a camera shows corrupted signals due to the car’s headlight. Qualitative results validate that our framework shows competitive detection performance in challenging environments.