GPA-3D: Geometry-aware Prototype Alignment for Unsupervised Domain Adaptive 3D Object Detection from Point Clouds

As mentioned in Sec. 3.2, for $i$-th point cloud scenario $P_{i}$ from the source or target domain, LiDAR-based detector generates the BEV features $\bm{F}_{i}\in\mathbb{R}^{H\times W\times C}$, where $H$, $W$, and $C$ denote the height, width and channel numbers of the feature map. We first project the corresponding ground truth $L^{s}_{i}$ or pseudo-labels $\hat{L}^{t}_{i}$ to the BEV feature map, and then randomly extract the equal-length sequences $\bm{F}_{i}^{+}\in\mathbb{R}^{M_{i}\times C}$ and $\bm{F}_{i}^{-}\in\mathbb{R}^{M_{i}\times C}$. Here, $M_{i}$ is the length of the feature sequence, $\bm{F}_{i}^{+}$ and $\bm{F}_{i}^{-}$ represent the foreground and background features from BEV, respectively.

For the extracted foreground features $\bm{F}_{i}^{+}$, we further divide them into different groups according to their geometric structures on point clouds. Specifically, for $j$-th foreground $\bm{F}_{i,j}^{+}$ in the sequence ($j\in[1,M_{i}]$), we compute its offset angle $\theta_{i,j}^{\text{off}}$ as follows:

where $r_{i,j}$ is the direction, $\theta_{i,j}^{\text{obs}}$ is the observation angle, as presented in Fig. 4 (left). Note that the direction $r_{i,j}$ is provided from the labels $L^{s}_{i}$ and $\hat{L}^{t}_{i}$, while the observation angle $\theta_{i,j}^{\text{obs}}$ can be computed according to the central position of 3D bounding box. Next, all foreground features are split into $K$ groups, and the group index $Q_{i,j}$ is formulated as:

where $norm(\cdot)$ is a normalization function that converting the input angles into $[0,2\pi]$, and $\delta=2\pi/K$ is the interval of angles between groups. In this way, the foreground features with similar offset angles $\theta_{i,j}^{\text{off}}$ are assigned into the same group, where their geometric structures are very similar, as demonstrated in Fig. 4 (right). Additionally, the extracted backgrounds $\bm{F}_{i,j}^{-}$ are sent into an individual group, thus totally $K+1$ groups are built.

At the beginning of training, we randomly initialize a series of learnable prototypes $\mathcal{G}=\{g_{k}\}_{k=1}^{K+1}\in\mathbb{R}^{(K+1)\times C}$. During training, we extract the BEV features $\bm{F}_{i}$ from both source and target domains, and split them into corresponding groups via Eq. 4. In $k$-th group, the foreground features $\bm{F}_{i,j}^{+}$ are enforced to be aligned with the foreground prototype $g_{k(k\in[1,K])}$. Similarly, the background features $\bm{F}_{i,j}^{-}$ in the last group are aligned with the background prototype $g_{K+1}$.

For the foreground features $\bm{F}_{i}^{+}$, we pull them closer with the corresponding prototype in $\mathcal{G}$, which can be formulated as:

where $sim(\bm{a},\bm{b})=\frac{\bm{a}\cdot\bm{b}}{||\bm{a}||\,||\bm{b}||}$ is the cosine similarity, $\mathbbm{1}[Q_{i,j}=k]$ is an indicator function that equals to 1 if $Q_{i,j}=k$ and 0 otherwise. Similarly, the background features $\bm{F}_{i}^{-}$ are also required to be pulled to the background prototype $\bm{g}_{K+1}$, which can be calculated as:

To enhance the discriminative capacity, we need to push the features away from all prototypes belonging to other groups. For example, the distances between background features $F_{i}^{-}$ and all foreground prototypes are minimized via:

For foreground features within adjacent groups, their corresponding geometric structures are relatively more similar. Repelling these features away is not very necessary, and might even make the training process unstable. Hence, we adopt a more relaxed constraints as follows:

where $m$ indicates the margin which is set to 0.5 in our experiments, $A_{i,j}$ is the index of the groups adjacent to $Q_{i,j}$, i.e., $A_{i,j}=Q_{i,j}\pm 1$. The overall soft contrast loss $\mathcal{L}_{contra}$ can be formulated as:

where $\beta_{1}$, $\beta_{2}$ and $\beta_{3}$ are the balance coefficients.

We evaluate the GPA-3D on widely used autonomous driving benchmarks including Waymo [30], nuScenes [2], and KITTI [6]. These datasets exhibit significant diversities in foreground patterns and LiDAR beams, which can lead to severe domain bias when transferring 3D detectors from one dataset to another. Detailed information about datasets is available in the supplementary material.

We verify the GPA-3D with two popular LiDAR-based detectors, namely SECOND-IoU [37] and PointPillars [16]. All the parameter settings for network architecture are set the same with OpenPCDet [33] and ST3D [37]. We perform all experiments using 8 NVIDIA V100 GPU cards. For the pre-training step, the model is trained for 30 epochs using the ADAM [15] optimizer and the total batch size of 32 on the source domain. Next, we utilize the pre-trained model to generate pseudo-labels on the target domain with a score threshold of 0.2. Note that instances with scores over 0.5 are retained and subsequently utilized to establish the high-quality pseudo-label database for IRA. Finally, we further fine-tune the model with our proposed approach for 30 epochs. To avoid local minima, we employ the cosine annealing strategy to adjust the learning rate, which was set to 0.003 for pre-training and 0.0015 for fine-tuning. Please refer to the supplements for more implementation details.

As shown in Tab. 2, GPA-3D is first compared with the Source Only method, which trains the model on the source domain and evaluates it on the target domain without any adaptation. Next, 5 existing works are included in the comparison, namely, SN [34], UMT [9], 3D-CoCo [40], ST3D [37], and ST3D++ [38]. SN utilizes the statistics from target annotations to normalize the foreground objects on the source domain. UMT employs a mean-teacher framework to filter inaccurate pseudo-labels. 3D-CoCo learns the instance-level transferable features for better generalization. ST3D and ST3D++ adopt a memory bank to produce high-quality pseudo-labels. Additionally, we also compare GPA-3D with the Oracle method, which trains the model on the labeled target data, serving as an upper bound for performance.

To validate the effectiveness to the domain shift about object size, we conduct a comprehensive comparison on Waymo $\rightarrow$ KITTI. As demonstrated in Tab. 1, with the 3D detector SECOND-IoU, our proposed GPA-3D outperforms ST3D++ [38] with a large margin, and significant performance gains are obtained compared with previous best results, i.e., 5.24% of AP${}_{\text{3D}}$ and 1.6% of AP${}_{\text{BEV}}$. Note that the AP${}_{\text{BEV}}$ of GPA-3D is also higher than Oracle method, indicating the effectiveness of incorporating the geometric structure information into UDA on 3D detection task. Even switching the base detector to PointPillars, our method still exceeds previous SOTA 3D-CoCo [40] by 7.94% and 1.19% in terms of AP${}_{\text{3D}}$ and AP${}_{\text{BEV}}$, respectively.

For the domain gap of LiDAR beams, we select Waymo $\rightarrow$ nuScenes as representatives due to their different LiDAR sensors, i.e., 64-beam vs 32-beam. As shown in Tab. 2, GPA-3D improves the adaptation performances to 37.25% AP${}_{\text{BEV}}$ and 22.54% AP${}_{\text{3D}}$ with the SECOND-IoU detector, surpassing previous SOTA methods. Compared with ST3D++ [38], 1.52% and 1.64% gains separately in terms of AP${}_{\text{BEV}}$ and AP${}_{\text{3D}}$ are achieved. Based on PointPillars, our approach exceeds the best method 3D-CoCo [40] by 2.37% in AP${}_{\text{BEV}}$, and outperforms ST3D [37] with 4.87% and 5.41% respectively in terms of AP${}_{\text{BEV}}$ and AP${}_{\text{3D}}$. These improvements demonstrate the advancement of our GPA-3D to mitigate the more challenging domain shift of cross-beam scenarios.

We assess the effectiveness of each component in GPA-3D, as presented in Tab. 3. Baseline (a) represents self-training via pseudo-labels on the target domain. The application of geometry-aware prototype alignment provides 5.92% and 2.62% gains separately in terms of AP${}_{\text{3D}}$ and AP${}_{\text{BEV}}$, and the soft contrast loss brings an improvement of 1.06% on AP${}_{\text{3D}}$. The improvements demonstrate that incorporating the geometric relationship into domain adaptation is feasible and effective. In addition, NSS and IRA boost the performance by around 2.5% and 1.5% respectively, which indicates the efficacy of enhancing the quality of supervision on target data.

We further investigate the effects of the geometry-aware prototype alignment. As illustrated in Fig. 7, the vanilla alignment with one pair of fore/background prototypes performs better than the co-training baseline, implying that the misalignment of features distribution affects the performance. Applying two prototypes yields 3.57% and 5.05% gains of AP${}_{\text{BEV}}$ and AP${}_{\text{3D}}$ respectively, compared to the co-training baseline. The performance reaches to the peak of 84.44% AP${}_{\text{BEV}}$ when 4 foreground prototypes are employed, indicating the advancement of combining geometric information with feature alignment. However, we observe minor performance degradation when too many prototypes are used, which we attribute to redundant prototypes leading to indistinguishable features in the representational space.

We conduct ablations on the noise sample filter (NSS) with various settings. As shown in Tab. 4, the detection performance drops to 67.79% AP${}_{\text{3D}}$, when we remove the NSS from GPA-3D. Only applying the NSS on target domain achieves the gains of 1.43% and 0.45% on AP${}_{\text{BEV}}$ and AP${}_{\text{3D}}$, respectively. We could see that using NSS on the source domain could also bring improvements. We think this is due to the fact that NSS suppresses those source samples with only a few points, which are very similar to the background noise. When the hard truncated $\alpha$ is adopted, AP${}_{\text{3D}}$ is further improved to 70.88%, indicating the effectiveness of NSS.

Also, we compare different policies in instance replacement augmentation (IRA). We can see from Tab. 5 that our proposed IRA attains 0.72% and 1.43% gains in terms of $\text{AP}_{\text{BEV}}$ and $\text{AP}_{\text{3D}}$, respectively. Without the group mechanism in IRA, i.e., randomly replacing pseudo-labels with instances the database, only marginal gains are obtained in $\text{AP}_{\text{3D}}$, and even degradation in $\text{AP}_{\text{BEV}}$. This highlights the significance of maintaining the consistency between instances and their contextual environments.

We compare our proposed GPA-3D with several adaptation frameworks, as presented in Tab. 6. The results confirm the effectiveness of GPA-3D, which leverages the geometric association to transfer 3D detectors across different domains. Fig. 8 further illustrates that, despite all models fluctuate at early epochs, our GPA-3D steadily and consistently enhances the detection performance in later training stages.

We exhibit some qualitative results of cross-domain adaptation in Fig. 6. Additionally, in Fig. 9, we visualize the distribution of BEV features. It is obvious that GPA-3D aggregates foreground samples into different prototypes, and separates them from the backgrounds. Further visualizations can be found in in the supplements.