Universal Domain Adaptive Object Detection via Dual Probabilistic Alignment

DAOD addresses the covariate shift from labeled data in the source domain to the unlabeled target domain under the closed-set categories. Existing DAOD methods can be categorized into adversarial training and mean teacher paradigms. As for adversarial training, DAF (Chen et al. 2018) incorporates global and instance alignment modules based on the Faster-RCNN detector, while incremental variants (Krishna, Ohashi, and Sinha 2023) improve global and instance alignment. ATMT (Li et al. 2023a) explores the potential of self-supervised learning, and EPM (Hsu et al. 2020) introduces a new FCOS detector. Li et al. propose graph-based alignment methods (Li et al. 2022; Li, Liu, and Yuan 2022b) to align class-conditional features, with IGG (Li et al. 2023b) enhancing graph generation by effectively addressing non-informative noise. As for the mean teacher paradigm, existing works (Chen et al. 2022b; Deng et al. 2023; Cao et al. 2023; Li, Guo, and Yuan 2023; Liu et al. 2022) focus on generating pseudo-labels for the target domain. In general, existing DAOD works have limited generalizability in open-world scenarios.

UniDA (You et al. 2019) is a general paradigm for partial-set (Zhang et al. 2018), open-set (Panareda Busto and Gall 2017), and closed-set domain adaptation (Tzeng et al. 2017). The existing UniDA can be categorized into four paradigms, including threshold, clustering, optimal migration, and mutual learning. The threshold methods (You et al. 2019; Fu et al. 2020; Chen et al. 2022a) estimate inter-sample uncertainty to identify shared categories, often relying heavily on manually set thresholds. Clustering-based UniDA methods (Saito et al. 2020; Li et al. 2021a) have been developed to distinguish shared categories. UniOT (Chang et al. 2022) introduces optimal transport to detect shared categories. Despite diversity advancements in classification tasks, UniDAOD remains in its early stages. US-DAF (Shi et al. 2022) conducts scale domain alignment and uses thresholds for sample mining. Recently, UCF (Lang et al. 2023) and W-adapt (Shi et al. 2024a) employ weighting mechanisms to mitigate the negative transfer of private categories. CODE (Shi et al. 2024b) leverages virtual domain labels to avoid domain-private samples alignment. In general, existing UniDAOD methods perform domain-shared category alignment at both the global and instance levels but overlook the alignment of domain-private categories.

To align global-level domain-private category features, we sample outliers in feature space and model the batch samples as Gauss distribution for the cumulative distribution function to estimate the domain distribution as the weights.

To efficiently align the domain-shared category at the instance level, we calculate the gradient norm of the instance samples to model a Gaussian distribution to discard outlier samples and estimate the weight of the domain-shared category samples to improve domain alignment.
Instance-level Sampling. The instance-level sampling process involves constructing the probability space and the sampling criteria. First, we build the probability space to model the Gaussian distribution. The number of $\hat{n}$ instance-level features $x_{\upsilon}$ is generated by the domain discriminator ${\mathcal{F}}_{\mathcal{D}_{\upsilon}}$ to calculate the domain probabilities $p_{\upsilon}={\rm Sigmoid}\left({\mathcal{F}}_{\mathcal{D}_{\upsilon}}(x_{\upsilon})\right)$. The domain probabilities are used to compute the gradient norm $\eta_{\upsilon}$ for each instance as $\eta_{\upsilon}=\left|p_{\upsilon}-{y_{d}}\right|$, where ${y_{d}}$ is the domain label. We then construct the gradient norm bins $\hat{\Omega}=\left\{i\cdot\psi|i\in\mathbb{Z}\right\}$ to calculate the gradient norm frequencies $\tau$, which model the Gaussian distribution with the minimum interval $\psi=\mathop{\rm argmin}\left\{(\eta_{\upsilon}^{\rm max}-\eta_{\upsilon}^{\rm min})\cdot\eta_{\upsilon}^{\rm std},\delta\right\}$, where $\delta$ is the hyperparameter. For the sampling criteria, we leverage the statistical characteristics of the Gaussian distribution. The first sampling criterion involves filtering out samples in noncontinuous frequency bins. The second criterion is related to the characteristic of the Gaussian distribution, where the frequency of filtered bins is lower than that of continuous bins. The sampling process is as follows:

where $\tau_{\omega}$ denotes the frequency in the first and last continuous bins in positive samples ${\rm\Omega}^{\rm pos}_{\upsilon}$. The sum of the bins represents the sampling radius from the feature centroid in the feature space, which dynamically adjusts the bins following a Gaussian distribution of the source or target domain data during adversarial training.
Calculating Alignment Weight. These negative instances are excluded from the instance alignment through the instance weight as follows:

where $\eta_{\upsilon}^{\rm mean}$ is the mean value of the gradient norm $\eta_{\upsilon}$.

Given the instance-level feature $x_{\upsilon}$ and the domain probabilities $p_{\upsilon}$, we first perform a classifier head to establish domain-private categories for both the source domain and the target domain: $\{{c}^{\ast}_{i}\mid\hat{y}_{s,i}^{{c}^{\ast}}\cap\hat{y}_{t,i}^{{c}^{\ast}}=\emptyset\}^{n^{*}}_{i}$. To aggregate the centroids of domain-private categories in feature and probability spaces, we calculate the feature centroid $\bar{x}_{\upsilon}$ and the probability centroid $\bar{p}_{\upsilon}$. We then conduct the cosine similarity distance $\mathcal{G}_{i}=\left\|x_{\upsilon,i}-\bar{x}_{\upsilon}\right\|_{2}$ and probability samples $\mathbf{g}_{i}=\left\|p_{\upsilon,i}-\bar{p}_{\upsilon}\right\|_{2}$ to centroid to measure intra-domain distances. To measure the intra-domain distance, we use cosine similarity, defined as $\varepsilon_{s/t}=\frac{1}{n^{*}}\sum_{i=1}^{n^{*}}\frac{\mathcal{G}_{i}\cdot\mathbf{g}_{i}}{\left\|\mathcal{G}_{i}\right\|_{2}\cdot\left\|\mathbf{g}_{i}\right\|_{2}}$. Finally, we employ the mean squared error (MSE) loss function to minimize the inter-domain distance, as follows:

The loss function ${\mathcal{L}_{\rm PCC}}$ optimizes the network by adopting a gradient detach for the source domain.

The training loss of the DPA is represented as ${\mathcal{L}}_{\rm DPA}$, which consists of the following loss terms:

where $\mathcal{L}_{\rm det}$ is the Faster-RCNN detector loss. $\mathcal{L}_{\rm GDPA}$ and $\mathcal{L}_{\rm IDSA}$ are the domain alignment losses for the GDPA and IDSA modules at the global and instance levels, respectively. The ${\mathcal{L}}_{\rm DPA}$ is optimized using the SGD optimizer. The bound loss ${\mathcal{L}_{\rm{bound}}}$ is optimized using the Adam optimizer with a learning rate set to $0.1$. The hyperparameter of $\alpha$ is 0 for the initial epoch and 0.1 thereafter.

We conduct extensive experiments following the setting (Shi et al. 2022) for three benchmarks: open-set, partial-set, and closed-set. The baseline approach (Chen et al. 2018) adopts Faster-RCNN as the base detector with the focal loss, and the backbone is ResNet-101 (He et al. 2016) or VGG-16 in (Simonyan and Zisserman 2014) pre-trained on ImageNet (Deng et al. 2009), which adopt VGG-16 in Cityscape to Foggy Cityscape and other benchmarks adopt the ResNet-101. The DPA model optimized training iterations are $100$k, with an initial learning rate of 1e-3 and a subsequent decay of the learning rate to 1e-4 following $50$k iterations. The detection performance is evaluated with the mean Average Precision (mAP) metric, and the threshold of mAP follows the setting (Shi et al. 2022) to set 0.5.

We evaluate our DPA framework on five datasets across three domain adaptation scenarios (open-set, partial-set, and closed-set): Foggy Cityscapes (Sakaridis, Dai, and Van Gool 2018), Cityscapes (Cordts et al. 2016), Pascal VOC (Everingham et al. 2010), Clipart1k (Inoue et al. 2018), and Watercolor (Inoue et al. 2018). In the open-set scenario, there are shared and private categories in both the source and target domains. We introduce mutil ratios $\beta=\left\{0.25,0.5,0.75\right\}$ to construct different shared category ratios $\beta=\frac{\mathcal{C}_{s}\cap\mathcal{C}_{t}}{\mathcal{C}_{s}\cup\mathcal{C}_{t}}$ benchmarks. In the partial-set scenario, the category set of the source domain is the subset for the target domain, and vice versa. In the closed-set scenario, the categories in the target and source domains are identical.

Open-set scenario. Tables 1, 2, and 3 present the open-set domain adaptive object detection performance from Pascal VOC to Clipart1k under different category overlap ratios $\beta$. The proposed framework consistently achieves state-of-the-art performance across various category settings compared to other methods. Compared to related DAOD methods (HTCN, MAF, DAF, SFA), the proposed DPA framework demonstrates significant performance advantages. Additionally, in comparison with UniDAOD methods (CODE, US-DAF, CMU, UAN, and UCF), the proposed DPA framework also exhibits superior performance.
Partial-set scenario. The results of the partial-set domain adaptation are presented in 4 and 5. In the partial-set scenario, the private categories are exclusively present in the source or target domain, leading to negative transfer. The proposed DPA method effectively addresses this issue and outperforms other approaches by a significant margin, achieving 54.1% and 56.3% mAP. For the domain-private category in the target domain ($C_{s}\subset C_{t}$), DPA enhances performance by 1.2% compared to the UniDAOD method (US-DAF), as shown in Table 4. For the domain-private category in the source domain (${C_{s}}\supset{C_{t}}$), DPA improves performance by 4.2% compared to the DAOD method (HTCN).
Closed-set scenario. The closed-set scenario results are shown in Table 6 show that although existing UniDAOD methods (Shi et al. 2024b, a) integrated with advanced DAOD methods achieve notable performance improvements on closed-set, the DAOD methods significantly outperform the UniDAOD methods. This advantage comes from DAOD under the closed-set assumption, while UniDAOD prioritizes dealing with label shifts in open environments. The proposed DPA framework exhibits satisfactory performance in closed-set scenarios through probability modeling.

We conduct ablation experiments on each submodule, with the corresponding results presented in Table 7. In these experiments, each module of the proposed DPA framework improves performance. The GDPA and IDSA modules provide significant gains when the domain-shared categories ratio is high ($\beta=50\%,75\%$), while the PCC module leads to more substantial improvements when the ratio is low ($\beta=25\%$).

To compare the performance of the proposed method with existing DAOD and UniDAOD methods in terms of positive and negative transfer, we present the performance gains of DAOD and UniDAOD relative to the source-only model in Fig. 3. The DAOD methods exhibit significant negative transfer, where DAF, MAF, and HTCN drop by approximately 2%, 4%, and 1% AP in class 0, respectively. In contrast, the UniDAOD methods mitigate negative transfer, with CODE and DPA achieving positive transfer of around 3% and 10% in class 4, respectively. This category-wise performance analysis proves that the proposed method effectively combats negative transfer and strengthens positive transfer.

We further analyze the probability gap in our DPA framework for open-set alignment. As shown in Fig. 4(a), the global-level mean probability gap is more pronounced in our DPA, highlighting its effectiveness in distinguishing domain-private categories. In contrast, Fig. 4(b) shows a smaller mean probability gap at the instance level, demonstrating that our DPA better aligns domain-shared categories. Additionally, we perform a weight quantitative analysis of global-level domain-private alignment, as illustrated in Fig. 5. As the ratio of domain-private categories increases, the mean weight gap also increases, indicating that adversarial training adaptively penalizes features associated with domain-private categories through weight adjustments.