Towards Robust Adaptive Object Detection under Noisy Annotations

Problem setting. In noisy DAOD, we are given noisy labeled source dataset $\mathcal{D}^{s}=\{x^{s}_{i},(y^{s},\tilde{y}^{s})_{i}\}_{i=1}^{N_{s}}$ and unlabeled target dataset $\mathcal{D}^{t}=\{x^{t}_{i}\}_{i=1}^{N_{t}}$. The labels $y^{s},\tilde{y}^{s}$ both contain box and class annotations, and source and target domains share an identical label space. However, the class annotations in $\tilde{y}^{s}$ contain noise. The objective is to learn a domain-adaptive object detector that can detects objects in $\mathcal{D}^{t}$.

How label noise affects domain adaptive object detection. For a domain adaptive detector, it attempts to learn a transformation $\upsilon$ that aligns the conditional distributions of image variable $X$ and label variable $Y$ from different domains, such that $P^{{t}}_{\upsilon(X^{{t}})\mid Y^{{t}}}=P^{{s}}_{\upsilon(X^{{s}})\mid Y^{{s}}}$. Then with the $P^{{s}}_{X^{{s}}Y^{{s}}}$ estimated from $\mathcal{D}^{s}$ and the distribution of the drawn samples from $P^{s}_{X^{s}}$, $P^{t}_{X^{t}}$, we are able to estimate $P^{t}_{Y^{t}}$, which is the marginal class distribution in the target domain. However, if the source dataset is noisy, then $P^{t}_{Y^{t}}$ is computed based on a biased joint distribution $\tilde{P}^{{s}}_{X^{{s}}\tilde{Y}^{{s}}}$ and the learned transformation $\tilde{\upsilon}$ will degrade the performance of domain adaptive object detection.

Concept of the proposed method. Although correcting all noisy labels in the source domain could fully recover the detection performance, it is unattainable practically and may not be optimal for achieving effective domain adaptation. To this end, we build a detection framework that jointly mines miss-annotated samples for recapturing missing semantics and explores the intrinsic positive impact on improving the generalization ability of the detector for class-corrupted samples rather than intuitively correcting the noisy annotations, which is illustrated in Fig. 3.

Intra-domain graph feature aggregation. Given proposals $\mathbf{P}\in\mathbb{R}^{N\times D}\leftarrow\{\mathbf{P},\overline{\mathbf{P}}\}$ after PIM, we first construct them as intra-domain undirected graphs $\mathcal{G}=\{\mathcal{V},\mathcal{E}\}$. Specifically, the vertices correspond to the proposals within each domain, and the edges are defined as the feature cosine similarity between them (${e}_{ii^{\prime}}=\frac{p_{i}\cdot p_{i^{\prime}}}{\parallel p_{i}\parallel_{2}\cdot\parallel p_{i^{\prime}}\parallel_{2}}$). Afterwards, we apply intra-domain aggregation to enhance the feature representation within each domain, shown as follows:

$${p}_{i}\leftarrow\sigma\Big{(}\sum_{i^{\prime}\in{\rm Neighbour}(i)}(w_{i}p_{i}{e}_{ii^{\prime}}+p_{i})\Big{)},\quad p_{i}\in\mathbf{P},\vspace{-6pt}$$

(2)

where the Neighbour($i$) denotes the proposals within the same domain as $i$, $\sigma$ is an activation function, and $w_{i}$ is a learnable weight that maps the original feature dimension $D$ to $D^{\prime}$. After the aggregation, proposals that share the common features could improve the feature representation and generate robust proposal features $\mathbf{P}$ with enhanced adaptive contexts within each domain.

Global relation matrix construction. To model the semantic relationship and effectively explore the category-wise transition probability between source and target domains, we introduce a global relation matrix that represents the category-wise affinity between domains. Specifically, considering the source dataset contains noisy annotations and the target dataset is unlabeled, we first utilize confident proposals after aggregation $\mathbf{P}$ to assemble as batch-wise prototypes, which correspond to the class-wise feature centroids:

$$\{\beta_{(u)}\}_{u=1}^{C}=\frac{1}{Card(\mathbf{P}_{(y^{\prime})})}\sum_{\begin{subarray}{c}y^{\prime}=u\\ p_{({y^{\prime}},i)}\in\mathbf{P}_{({y})}\end{subarray}}p_{(y^{\prime},i)},\vspace{-7pt}$$

(3)

where $Card$ is the cardinality and $y^{\prime}$ is the most confident category. Then, the correspondence between local and global prototypes is characterized according to their semantic correlation, and an adaptive update operation for generating global prototypes $\{\mathcal{B}^{s}_{(u)}\}_{u=1}^{C}$ and $\{\mathcal{B}^{t}_{(v)}\}_{v=1}^{C}$ is conducted:

$$\{\mathcal{B}_{(u)}\}_{u=1}^{C}=\sum_{m=1}^{C}(1-\tau_{(m,u)})\beta_{(m)}+\tau_{(m,u)}\mathcal{B}_{(u)},\vspace{-7pt}$$

(4)

where $\tau_{(m,u)}$ is the cosine similarity between the $m$-th batch-wise prototype and the $u$-th global prototype. With this adaptive update process, the representation of global prototypes $\{\mathcal{B}^{s}_{(u)}\}_{u=1}^{C}$ and $\{\mathcal{B}^{t}_{(v)}\}_{v=1}^{C}$ can be strengthened via robust and compact batch-wise local features. Finally, the global relation matrix $\Pi\in\mathbb{R}^{C\times C}$ is constructed by the cosine similarity between the prototypes, where each entry $\pi_{u,v}$ represents the affinity between the $u$-th prototype in the source domain and the $v$-th prototype in the target domain.

Transition probability regularization. During alignment, the class-wise domain knowledge of class-corrupted samples are inequilibrium with correctly-labeled samples. To mitigate this, the transition probabilities of the class-corrupted samples are expected to be regularized by the intrinsic class-wise correspondence between source and target domain. Therefore, we directly extract noisy source proposals features from $\tilde{\mathbf{P}}^{s}_{(\tilde{y})}$ regarding to their corresponding noisy labels $\tilde{y}$, and generate noisy source local prototype similar to the batch-wise prototypes in Eq. (3):

$$\{\tilde{\beta}^{s}_{(u)}\}_{u=1}^{C}=\frac{1}{Card(\tilde{\mathbf{P}}^{s}_{(\tilde{y})})}\sum_{\begin{subarray}{c}\tilde{y}=u\\ \tilde{p}^{s}_{(\tilde{y},i)}\in\tilde{\mathbf{P}}^{s}_{(\tilde{y})}\end{subarray}}\tilde{p}^{s}_{(\tilde{y},i)}.\vspace{-7pt}$$

(5)

Then, we build local relation matrix $\mathbf{Z}\in\mathbb{R}^{C\times C}$ between $\tilde{\beta}^{s}_{(u)}$ and $\mathcal{B}^{t}_{(v)}$ to model the transferability of noisy source samples. Each entry is the class-wise transition probability ${z}_{u,v}=\frac{\tilde{\beta}^{s}_{(u)}\cdot\mathcal{B}^{t}_{(v)}}{\parallel\tilde{\beta}^{s}_{(u)}\parallel_{2}\cdot\parallel\mathcal{B}^{t}_{(v)}\parallel_{2}}$. We use $\ell$1 loss to regularize such transition probability between local relation matrix and global relation matrix:

$$\mathcal{L}_{mgrm}=\frac{1}{r}\sum_{r\in\mathds{1}(\mathbf{Z})}|{z}_{r}-{\pi}_{r}|,\vspace{-6pt}$$

(6)

where $\mathds{1}(\mathbf{Z})$ refers to the non-zero columns within $\mathbf{Z}$, which indicates the existence of the $r$-th category within the batch. Different from other methods that build category-wise graphs [58, 44] or maintain batch-wise graphs with fixed shapes [49] to model the relationship between source and target domains, our proposed MGRM combines the semantic knowledge between source and target domains, and use it to regularize the transition probability of noisy features implicitly. Therefore, the transferable representations of feasible adapted noisy samples can be extracted for achieving effective semantic alignment.

Entropy-aware alignment. To alleviate the performance deterioration on the target domain, the domain adaptive detector conducts a min-max game to yield a saddle-point solution $\left(\hat{\phi}_{f},\hat{\phi}_{det},\hat{\phi}_{dis}\right)$:

	$\displaystyle\left(\hat{\phi}_{f},\hat{\phi}_{det}\right)$	$\displaystyle=\arg\min_{{\phi}_{f},{\phi}_{det}}\mathcal{L}_{det}-\mathcal{L}_{dis},$		(7)
	$\displaystyle\left(\hat{\phi}_{dis}\right)$	$\displaystyle=\arg\min_{{\phi}_{dis}}\mathcal{L}_{dis},$

where $\hat{\phi}_{f}$, $\hat{\phi}_{det}$, and $\hat{\phi}_{dis}$ refer to the optimal parameters of the feature extractor, the detector, and the discriminator respectively, which compose the entire domain adaptive detection framework $\hat{\phi}$. However, the noisy labels in the source domain will cause an incompatible optimation between $\left({\phi}_{f},{\phi}_{det}\right)$ and $\left({\phi}_{dis}\right)$ as the discriminator is class-agnostic, resulting in an insufficient upper boundary of the source risk [28, 43]. A natural solution is to directly map category onto features for discrimination [57, 28], but it could further magnify the effect of noisy labels under the noisy DAOD setting if the detector is biased. Hence, we build an entropy-aware discriminator to alleviate this effect. Specifically, for each source and target proposal feature $p^{s}_{i}\in\mathbf{P}^{s},p^{t}_{i}\in\mathbf{P}^{t}$ and their corresponding logits $\eta^{s}_{i}\in\boldsymbol{\eta}^{s},\eta^{t}_{i}\in\boldsymbol{\eta}^{t}$ generated with $\phi_{det}$, we concatenate them and feed into a discriminator $\phi_{dis}^{EAGR}$, as shown in Fig. 3(c). The loss function of the discriminator is written as:

	$\displaystyle\mathcal{L}_{dis}^{EAGR}=$	$\displaystyle-\sum_{i,j}zlog(\phi_{dis}^{EAGR}(p^{s}_{i}\text{ⓒ}\eta^{s}_{i}))+$		(8)
		$\displaystyle(1-z)log(\phi_{dis}^{EAGR}(p^{t}_{j}\text{ⓒ}\eta^{t}_{j})),$

where $z$ refers to the domain label, which is 1 for source and 0 for target. Considering the entropy criterion $H(\eta)=-\sum_{u=1}^{C}\eta_{u}\log(\eta_{u})$ that quantifies the uncertainty of classifier predictions, the concatenated logits are softly conditioned on the pooled RoI features [31, 21, 34] to implicitly associate each instance to several most related categories. Hence, category information is preserved for discriminators in aligning class-wise semantic features within each domain, meanwhile providing entropy-aware gradients for the subsequent gradient concilement process.

Gradient reconcilement. Given a domain adaptive detector with parameters $\phi$ and objective function $\mathcal{L}$, we have gradients for different roles of the proposals:

	$\displaystyle G^{s}_{cln}$	$\displaystyle=\mathbb{E}_{x^{s}\in D^{s}_{cln}}\frac{\partial\mathcal{L}[(x^{s},y^{s});\phi]}{\partial\phi},$		(9)
	$\displaystyle G^{s}_{cpt}$	$\displaystyle=\mathbb{E}_{x^{s}\in D^{s}_{cpt}}\frac{\partial\mathcal{L}[(x^{s},\tilde{y}^{s});\phi]}{\partial\phi},$
	$\displaystyle G^{t}$	$\displaystyle=\mathbb{E}_{x^{t}\in D^{t}}\frac{\partial\mathcal{L}[(x^{t});\phi]}{\partial\phi},$

where $G^{s}_{cln}$, $G^{s}_{cpt}$, $G^{t}$ are the gradients provided by clean proposals, noisy proposals, and target proposals, respectively. After the entropy-aware alignment, the gradients $G^{s}_{cln}$, $G^{s}_{cpt}$, and $G^{t}$ are conditioned to the class-wise information provided by both noisy labels and the entropy of classifier predictions. Considering that both $G^{s}_{cln}$ and $G^{t}$ optimize the feature extractor $\hat{\phi}_{f}$ and detector $\hat{\phi}_{det}$ in the direction towards learning domain-invariant representations, the value of their inner product $G^{s}_{cln}\cdot G^{t}$ is expected to be sufficiently large. Nevertheless, the direction of $G^{s}_{cpt}$ could not be determined as they are produced by noisy samples. To reliably characterize the domain-invariant portion within $G^{s}_{cpt}$, we simultaneously maximize $G^{s}_{cpt}\cdot G^{s}_{cln}$ and $G^{s}_{cpt}\cdot G^{t}$ to encourage the direction of gradients provided by noisy and clean samples to be consistent. Thus, we are expected to maximize the summation of the above terms:

$$\arg\max_{{\phi}_{f},{\phi}_{det},{\phi}_{dis}}\big{(}G^{s}_{cln}\cdot G^{s}_{cpt}+G^{s}_{cln}\cdot G^{t}+G^{s}_{cpt}\cdot G^{t}\big{)}.\vspace{-4pt}$$

(10)

As we cannot directly optimize Eq. (10) with SGD as clean and noisy gradients cannot be explicitly split, meanwhile computing the Hessians (second order derivatives) is computational prohibitive, inspired by [33, 41], we utilize the first-order meta update of the network as an approximation, which could maximize the above inner product between gradients over iterations and avoid splitting $G^{s}_{cln}$ and $G^{s}_{cpt}$ :

$$({\phi}_{f},{\phi}_{det})\leftarrow({\phi}_{f},{\phi}_{det})+\lambda(\Delta\tilde{\phi}_{f},\Delta\tilde{\phi}_{det}),$$

(11)

where $(\Delta\tilde{\phi}_{f},\Delta\tilde{\phi}_{det})$ denotes the residual of the parameters before and after multi-step training of the network and $\lambda$ is the meta weight. The detailed proof of the approximation is provided in the supplementary. With EAGR, the semantic and discrimination information are harmonized into the backpropagation process, and the gradients of distinct samples are encouraged to achieve coherence. Therefore, both clean and noisy samples would be contributive towards learning a domain-invariant object detector.

Pascal VOC & Noisy Pascal VOC $\to$ Clipart1k. We list the results of using Pascal VOC and Noisy Pascal VOC with different noisy rates as the source domain, and Clipart1k as the target domain in Table 1. It is shown that the performance of the baseline domain adaptive detector [6] drops consistently as the noisy rate increases, and drops from $35.0\%$ to $28.5\%$ under $80\%$ noisy annotations. With loss adjustment method CP [35], the performance shows limited improvement and even drops by $0.5\%$ and $2.0\%$ at $20\%$ and $40\%$ noisy rates. With symmetric loss methods SCE [45] and GCE [56], the detector performs better than CP under noisy settings, but they significantly deteriorate the performance of the detector under clean scenario, causing the mAP drops by $2.5\%$ and $1.8\%$, respectively. However, adding our proposed NLTE not only achieves robust adaptive detection under different noisy rates ($2.3\%$ mAP improvement at $20\%$ and $4.0\%$ mAP improvement at $80\%$), but also guarantees the performance of the domain adaptive detector when the source annotations are clean.

Pascal VOC & Noisy Pascal VOC $\to$ Watercolor2k. Table 2 shows the experimental results of Pascal VOC and Noisy Pascal VOC $\to$ Watercolor2k. In the clean setting, adding all compared methods [45, 35, 56] perform worse than the baseline DAF [6], indicating that they could suffer from underfitting and hurt the detection performance. While the proposed NLTE improves the mAP by $1.3\%$, which is attributed to its capacity of promoting the generalization ability through fully utilizing the noisy samples instead of correcting them straightforwardly. As the noise rate increases from $20\%$ to $80\%$, adopting NLTE consistently improves the mAP by $2.8\%$, $6.8\%$, $8.4\%$, and $5.5\%$, respectively, while the compared methods show unstable improvements. The results evidently suggest that NLTE efficiently boosts the robustness of domain adaptive object detectors.

Cityscapes $\to$ Foggy Cityscapes. We consider Cityscapes as a real-world noisy annotated source dataset for DAOD and directly implement DAF+NLTE in the Cityscapes $\to$ Foggy Cityscapes benchmark to validate its effectiveness. As listed in Table 3, DAF+NLTE shows promising improvements over other state-of-the-arts that were tailored for the DAOD task with the same (or less) training epochs and under the same settings. Specially, with larger training and testing scales as EPM [19] and SSAL [32], i.e., setting the short side of the images to 800 pixels, DAF+NLTE achieves an mAP of $45.4\%$, outperforming SSAL by $5.8\%$. The results indicate that existing DAOD methods may suffer from biased source data and addressing the noisy annotations is arguably important for achieving effective adaptation.