Relation Matters: Foreground-aware Graph-based Relational Reasoning for Domain Adaptive Object Detection

Object detection, one of the most longstanding and challenging problems in computer vision, aims at localizing and classifying all object instances of given categories in natural images [61]. The current state-of-the-art is shaped by deep learning based approaches. This paper focuses on exploring the adaptability of modern object detectors, hence we briefly review a few representative object detectors. The series of region-based convolutional networks, such as R-CNN [62], Fast R-CNN [63], and Faster R-CNN [27], have demonstrated strong capability in achieving higher detection precision. These methods first get a set of object proposal candidates generated by the region proposal mechanisms, and then the proposals are further classified and their locations are refined by regression in the second stage. The other mainstream pipeline is one-stage detector, such as SSD [28], YOLO [64, 65], and RetinaNet [66], which directly conducts the category confidence prediction and the bounding box regression in a single-shot manner. By doing so, these detectors have shown a clear superiority in terms of inference speed. In addition, anchor-free one-stage object detectors, such as CornerNet [67] and FCOS [68], eliminate the need for anchor boxes and thus result in a much simpler training process.

DAOD [69, 70] aims to mitigate the distributional shift problem in object detection. Domain Adaptive Faster R-CNN [16] pioneers this line of research by adversarially learning domain-invariant representations on both image-level and instance-level within the Faster R-CNN framework. In view of the local nature of object detection task, most existing DAOD methods [17, 18, 19, 20, 21, 22, 23, 24, 25, 71, 72, 73, 74, 26, 75, 76] resort to learning the local feature patterns in the target domain in virtue of elaborate fine-grained feature alignment modules.

Specifically, Zhu et al. [17] design a region mining strategy to identify the discriminative regions, and then utilize the source region proposals to reweight the target ones to induce better local alignment; Xu et al. [22] and Zheng et al. [23] rely on the cross-domain prototype alignment [13, 14] to align the foreground objects with the same class label across domains. He et al. [77] propose an asymmetric tri-way Faster-RCNN to solve the labeling unfairness between domains. Wu et al. [26] explicitly extract domain-invariant instance-level features based on a progressive disentangled mechanism. Wang et al. [78] and Chen et al. [79] additionally incorporate the object scale into adversarial training networks. Hsu et al. [25] and VS et al. [74] introduce specific networks to generate foreground-aware attention maps, which are used to achieve category-wise adversarial feature alignment. Although our work have the similar motivation about creating foreground-background awareness compared to some of local alignment methods, these prior efforts still focus on pairwise alignment and usually require additional attention modules.

In addition to the series of works that adapt Faster R-CNN, Kim et al. [80] aims to adapt SSD via weak self-training and adversarial background score regularization. DA-DETR [81] develops a hybrid attention module with a domain discriminator on the top of Deformable DETR [82] to highlight and align the foreground features in adversarial training. Transformer-based methods [83] globally model the long-range dependencies among all encoded features while our FGRR locally models the relations based on the constructed graphs (a sparse version of encoded features). Here, the difference between global and local is that whether the self-attention operation is applied to all encoded features.

Despite their strong efficacy on adapting certain detectors, state-of-the-art methods cannot be readily applied to distinct detection pipelines and thus fail to establish a general DAOD approach. More importantly, how to effectively perform inter- and intra-domain relational reasoning to capture the topological relationships among foreground objects is vital for enhancing the discriminability of feature representations in DAOD but remains the boundary to explore.

Assume that we are given the source and target 3D feature maps $F_{s},F_{t}\in\mathbb{R}^{C\times{H}\times{W}}$ extracted from shallow layer of the backbone network, i.e., the extracted features are low-level features. First of all, we aim to find source foreground pixels by using source annotations, including category labels and bounding boxes. In object detection, the semantic label of a certain source pixel is determined by the scope of its bounding box, which inevitably includes many background pixels. Fortunately, we can observe that the foreground pixels are meaningful and coherent compared to the uninformative background content within the bounding boxes.

Technically, we first compute the centroid for each source category in the feature space, which represents the mean feature vector of pixels belonging to the same object category within a source image. The formulation is defined as follows,

where $i$ is the pixel index, and $I_{s}^{k}$ denotes the set of pixels annotated with category $k$ in $F_{s}$. Then, we utilize the calculated $c_{s}^{k}$ to select foreground pixels in $I_{s}^{k}$ that satisfy the following two requirements:

• •

  For each source pixel $F_{s}^{i}$ in $I_{s}^{k}$, its similarity to the class centroid $c_{s}^{k}$ should larger than a certain value, i.e., $\cos(F_{s}^{i},c_{s}^{k})>\tau_{1}$, where $\cos(\cdot,\cdot)$ denotes the cosine similarity and $\tau_{1}$ is a threshold.

• •

  To mitigate the influence of pixels that are located near the bounding box, the centerness [68] of pixel point $F_{s}^{i}$ should also larger than a certain value, i.e., $\sqrt{\frac{\min(l,r)}{\max(l,r)}\times\frac{\min(t,b)}{\max(t,b)}}>\tau_{2}$, where $l$, $r$, $t$, and $b$ denote the normalized distances from the location of $F_{s}^{i}$ to the four sides of the bounding box, and $\tau_{2}$ is a threshold.

If the above requirements are all satisfied, $F_{s}^{i}$ is added into $\hat{I}_{s}^{k}$, where $\hat{I}_{s}^{k}$ is the selected foreground pixel set annotated with class $k$. The overall source selected foreground pixel set is denoted by $\hat{I}_{s}=\{\hat{I}_{s}^{k}\}_{k=1}^{K}$.

Next, we aim to find pixels in $F_{t}$ that are more likely to be foreground pixels via cross-domain nearest neighbor search. We present the searching process below. For a source pixel $i$ in $\hat{I}_{s}^{k}$, we searched its nearest neighbor $j^{\prime}$ in the target domain. In turn, $i^{\prime}$ is the nearest neighbor of target pixel $j^{\prime}$ in the source domain. If $i^{\prime}$ also belongs to the category $k$, we will assign the target pixel $j^{\prime}$ with pseudo-label $k$. Similarly, we traversal all pixels in $\hat{I}_{s}^{k}$ to find out pixel pairs between domains that are mutual nearest neighbors. In this way, we can obtain two set of selected pixels in both domains, i.e., $\hat{I}_{s}$ and $\hat{I}_{t}$.

Given the searched foreground pixels in both source and target domains, we can easily project these visual features into node space, where each node corresponds to a foreground pixel in the feature map. Then, we aim to learn inter-domain visual dependencies in pixel level by introducing a bipartite graph convolutional module (BGCM) that can be inserted to detection model in a plug-and-play manner. To be specific, BGCM utilize a bipartite graph $\mathcal{G}_{\rm inter}^{P}=(\mathcal{V}_{s}^{P},\mathcal{V}_{t}^{P},\mathcal{E}^{P})$ to indicate the relations among foreground nodes across domains and enhance their expressive power by aggregating information from its neighborhoods on the opposite domain. $\mathcal{V}_{s}^{P}$ and $\mathcal{V}_{t}^{P}$ are two group of bipartite graph nodes projected from $\hat{I}_{s}$ and $\hat{I}_{t}$ respectively. Bipartite graph edges $\mathcal{E}^{P}$ represent the similarities between $\mathcal{V}_{s}^{P}$ and $\mathcal{V}_{t}^{P}$. To mitigate the impact of noisy background pixels and relations, we let the edge weights be learnable. Formally, for nodes $i$ and $j$ in $\mathcal{V}_{s}^{P}$ and $\mathcal{V}_{t}^{P}$, we have,

where $\sigma$ stands for the sigmoid function, $F_{s}^{i}$ and $F_{t}^{j}$ are the features of nodes $i$ and $j$, and $\theta_{e}^{P}$ is the learnable parameter.

To perform graph convolution on the constructed bipartite graph $\mathcal{G}_{P}$, we augment its original form as follows,

After that, the augmented bipartite graph $\hat{\mathcal{G}}_{\rm inter}^{P}=\{\hat{\mathcal{V}}^{P},\hat{\mathcal{E}}^{P}\}$ can be learned by leveraging the modern Graph Convolutional Networks (GCN) [86]. In practice, we stack multiple graph convolution layers to form the BGCM. To be specific, the graph convolution is recursively conducted as: ${\bm{X}}^{(l+1)}={\rm ReLU}\left(\hat{{\bm{A}}}{\bm{X}}^{(l)}{\bm{W}}^{(l)}\right)$, where ${\bm{W}}^{l}$ is the parameter matrix, ${\bm{X}}^{l}$ are the hidden features of the $l$-th layer (where $1\leq{l}\leq{L}$), and $\hat{{\bm{A}}}$ is the adjacency matrix.

To improve the discriminability of node features, we conduct node classification (i.e. multi-class classification) based on the bipartite graph. Note that the selected source pixels have ground-truth labels, and we utilize the pixel-wise correspondence to assign pseudo-labels to the selected target pixels. Formally, the last layer of pixel-level bipartite graph (BGCM) predicts the label using a classifier and can be formulated as follows,

where $\hat{y}$ denotes the predicted label, $FC$ denotes a fully-connected layer, and $x$ is the feature of source or target nodes. The node classification loss is represented by $\mathcal{L}_{\rm NC}$.

Although BGCM models the correlations between domains, the intra-domain topological structure is neglected. We argue that relations within per domain are complementary to the inter-domain relations and should be explicitly taken into consideration. To solve this issue, we introduce a graph attention module (GAM), which allows nodes to attend over their neighborhoods features by specifying different weights to different neighbors, to perform intra-domain relational reasoning and capture the long-range dependencies between distant regions on each input image.

Specifically, we directly construct the pixel-level intra-domain graph $\mathcal{G}_{\rm intra}^{P}=\{\mathcal{V}^{P},\mathcal{E}_{\rm intra}^{P}\}$ on the source or target selected pixels ($\hat{I}_{s}$ or $\hat{I}_{t}$) to reduce the computational cost. Each node in $\mathcal{V}^{P}$ represents a selected pixel, and each edge in $\mathcal{E}_{\rm intra}^{P}$ encodes the relationships between nodes. Then, we conduct self-attention on the nodes [87], which computes the hidden representations that characterize the relationships between a source/target selected pixel and its neighborhoods within the domain. Given two nodes $i$ and $j$ in $\mathcal{G}_{\rm intra}^{P}$, their edge weights are defined as follows,

where $\mathcal{N}_{i}$ stands for the neighborhood of node $i$ in the graph, ${\bm{a}}$ is a learnable weight vector, ${\bm{W}}$ is a weight matrix that is applied to every node, and $\|$ is the concatenation operation.

Discussion. Technically, graph self-attention is a type of local attention, while self-attention in transformer [83] is a type of global attention. The constructed intra-domain relation graphs are sparse in the feature space. In that sense, local attention (based on the constructed graphs) will significantly reduce the potential of being biased towards those non-transferable nodes (pixels or regions), while global attention (based on the complete feature maps) is more likely to result in wrong feature aggregation and even negative transfer.

Semantic-level graph nodes are introduced to represent the inherent characteristics of high-level visual concepts (object/instance/category). Regarding semantic-level graph, an intuitive idea is to utilize the proposals generated by RPN as graph nodes. Unfortunately, these proposals may be insufficient to stand for an instance and cannot directly feed them into the relational reasoning procedure in view of the incomplete categorical information problems in the target domain, such as the deviation of bounding boxes, occlusions, and class ambiguities. Moreover, due to the absence of target ground-truth bounding boxes, most existing DAOD works randomly sample target candidate proposals for ROI-pooling and fail to distinguish the positive and negative samples. To solve this problem, we propose to use pseudo-labels for discriminating the target foreground instance-level features.

The workflow of Faster R-CNN can be divided into three components: backbone network, region proposal network (RPN), and ROI-wise classifier (RC). RPN generates and sends candidate object proposals to the head of RC for ROI-pooling based on the feature map extracted from the backbone network, then RC predicts the category labels and box coordinates of the pooled instance-level features. Both RPN and RC include a classification loss and a regression loss. Specifically, we utilize the detection results of RC to generate pseudo-labels for the target proposals. At the second stage, assume that we have $N_{t}$ target proposals filtered by RPN. The classification and regression outputs of RC are denoted by $F_{cls}(\bm{f}_{p}^{i})$ and $B_{p}^{i}$, where $i=\{1,2,\dots,N_{t}\}$ and $\bm{f}_{p}^{i}$ represents the proposal feature. If $k^{\prime}=\arg\max\limits_{k}\;F_{cls}(\bm{f}_{p}^{i})$, this result will be added into class $k^{\prime}$. Then, we sort the proposals within each class in descending order by their classification probabilities. A portion of high-confidence detection results in each class are selected as our pseudo labels, which are denoted by $\hat{B}_{p}^{i}$. In this way, we can ensure that the sampling process works in a class-balanced manner, implicitly highlighting the under-represented categories. Finally, we utilize the pseudo-labeled bounding boxes to obtain corresponding proposal features from RPN, i.e., $\bm{\hat{f}}_{p}^{i}={\rm\text{ROI-pooling}}(\bm{f}_{g},\hat{B}_{p}^{i})$. These discriminative ROI-based instance-level features are the representations of semantic-level graph nodes, and the constructed graph can regarded be sparse representation of all region proposals. Note that the target negative samples (backgrounds) are still randomly sampled from the remaining proposals.

Next, we propose a semantic-level bipartite graph module (SBGM) to model semantic relations and constraints with respect to the foreground objects across domains. The bipartite graph is defined as $\mathcal{G}_{\rm inter}^{S}=\{\mathcal{V}_{s}^{S},\mathcal{V}_{t}^{S},\mathcal{E}^{S}\}$. $\mathcal{V}_{s}^{S}=\{v_{s_{i}}\}_{i=1}^{N_{p}}\in\mathbb{R}^{N_{p}\times{d}}$ and $\mathcal{V}_{t}^{S}=\{v_{t_{j}}\}_{j=1}^{N_{p}}\in\mathbb{R}^{N_{p}\times{d}}$ are the source and target vertex sets, where $v_{s_{i}}$ and $v_{t_{j}}$ is the ROI-based instance-level features generated by RPN, $N_{p}$ is the number of proposals, $d$ is the node feature dimension, and $\mathcal{E}^{S}$ denotes the set of edges.

Given the graph nodes, SBGM aims to characterize the correspondence between $\mathcal{V}_{s}^{S}$ and $\mathcal{V}_{t}^{S}$. A natural idea is to traverse all possible pairs between $\mathcal{V}_{s}^{S}$ and $\mathcal{V}_{t}^{S}$ to compute their similarity, and node pairs with higher similarity should be assigned larger edge weights. Whilst we have enhanced the discriminability of foreground proposals, the instance-level features still contain positive and negative samples, and the distinction between known and unknown classes is yet to be explicitly constrained in light of the asymmetry of OSDA problem, making the target nodes be risky to aggregate biased or even wrong semantic information during the message-passing process.

Consequently, to identify and isolate the background proposals, we design a Cross-Domain Similarity Regularization (CDSR) strategy to generate reliable node pairs across domains. Our key idea is to adjust the cross-domain similarity measure to ensure that the nearest neighbor of a source node, in the target domain, is more likely to have as a nearest neighbor this particular source node, i.e., assign large edge weights to nodes from $\mathcal{V}_{s}^{S}$ and $\mathcal{V}_{t}^{S}$ that are mutual nearest neighbors. However, compared to the low-level feature space, we found that the nearest neighbors may be asymmetric in the high-level embedding space: $v_{t}$ (we omit the subscripts $i$ and $j$ for simplicity) being a $K$-NN of $v_{s}$ does not assure that $v_{s}$ is a $K$-NN of $v_{t}$, which also refer to the hubness problem [88, 89]. In high-level semantic space, some nodes are more likely to be the nearest neighbors of many other nodes (e.g., easy negatives), but some others may be not nearest neighbors of any node (e.g., hard positives). For the bipartite graph $\mathcal{G}_{\rm inter}^{S}$, we denote the neighborhood of a source node $v_{s}$ as $\mathcal{N}_{T}(v_{s})$. All $K$ elements of $\mathcal{N}_{T}(v_{s})$ are nodes from $\mathcal{V}_{t}^{S}$. Likewise, the neighborhood associated with a target node $v_{t}$ is represented by $\mathcal{N}_{S}(v_{t})$. All $K$ elements of $\mathcal{N}_{S}(v_{t})$ are nodes from $\mathcal{V}_{s}^{S}$. The average similarity of a source node $v_{s}$ to its target neighborhood is represented by,

Similarly, the average similarity of a target node $v_{t}$ to its source neighborhood is represented by $r_{S}(v_{t})$. Based on the computed $r_{T}(v_{s})$ and $r_{S}(v_{t})$, the cross-domain similarity measure ${\rm CDSR}(\cdot,\cdot)$ is formulated as follows,

Then, we can utilize Eq. (8) to calculate the adjacency matrix ${\bm{A}}$, which associates each edge $(v_{s_{i}},v_{t_{j}})$ with its element ${\bm{A}}_{ij}$. Finally, we use Eq. (3)-(4) to augment $\mathcal{G}_{\rm inter}^{S}$ as $\hat{\mathcal{G}}_{\rm inter}^{S}=\{\hat{\mathcal{V}}^{S},\hat{\mathcal{E}}^{S}\}$ such that the modern graph convolution techniques can be conducted based on $\hat{\mathcal{G}}_{\rm inter}^{S}$.

To further enhance the semantic correlations among foreground objects between domains, we propose a Category-aware Domain Alignment (CDA) loss term on top of $\hat{\mathcal{G}}_{\rm inter}^{S}$ to enable domain alignment on all foreground categories. Technically, we contrastively align the source and target prototypes to achieve domain alignment. Formally, we define the source and target class prototypes as follows,

where $\hat{\mathcal{G}}_{k}^{S}$ stands for the nodes in $\mathcal{G}_{\rm inter}^{S}$ that belongs to category $k$ ($k\in\{1,2,\dots,K\}$). The target graph nodes can be clustered into $K$ classes by using the target pseudo-labels. Formally, the formulation of CDA loss is defined as follows,

where $\xi$ is the margin term and set as 1 in our experiments.

The semantic-level intra-domain graph are represented by $\mathcal{G}_{\rm intra}^{S}=\{\mathcal{V}^{S},\mathcal{E}_{\rm intra}^{S}\}$. The graph nodes are inherited from the SBGM. Each edge in $\mathcal{E}_{\rm intra}^{S}$ stands for the semantic correlations between $v_{i}$ and $v_{j}$. To derive the intra-domain graph edge representations, we regularize the semantic relation from two aspects, i.e., geometric constraints and semantic similarities. Specifically, intra-domain graph edge is represented by an adjacency matrix ${\bm{A}}\in\mathbb{R}^{|\mathcal{V}^{S}|\times|\mathcal{V}^{S}|}$, which includes a spatial graph and a semantic graph.

For the spatial graph, we introduce $dist(B_{p}^{i},B_{p}^{j})$ and $\text{IoU}(B_{p}^{i},B_{p}^{j})$ to characterize the spatial interactions between nodes, where $B_{p}^{i}$ and $B_{p}^{j}$ are two region proposals associated with two RoI-pooled instance-level features, $dist(B_{p}^{i},B_{p}^{j})$ denotes the distance between $B_{p}^{i}$ and $B_{p}^{j}$, and $\text{IoU}(B_{p}^{i},B_{p}^{j})$ measures their interactions in terms of Intersection over Union (IoU). Formally, when $dist(B_{p}^{i},B_{p}^{j})<0.5$ or $\text{IoU}(B_{p}^{i},B_{p}^{j})>0.5$, $B_{p}^{i}$ and $B_{p}^{j}$ will be connected with a spatial graph edge (i.e. ${\bm{A}}_{ij}^{\rm spt}=1$); otherwise, these two nodes will not be connected (${\bm{A}}_{ij}^{\rm spt}=0$). By doing so, we can define the spatial adjacency matrix ${\bm{A}}^{\rm spt}$.

For the semantic graph, we directly measure their cosine similarity to characterize the semantic interactions between nodes. when $\cos(B_{p}^{i},B_{p}^{j})>0.5$, $B_{p}^{i}$ and $B_{p}^{j}$ will be connected with a semantic graph edge (i.e. ${\bm{A}}_{ij}^{\rm sec}=1$); otherwise, these two nodes will not be connected (${\bm{A}}_{ij}^{\rm sec}=0$). By doing so, we can define the spatial adjacency matrix ${\bm{A}}^{\rm sec}$.

We integrate the spatial graph with the semantic graph to formulated our complete semantic-level intra-domain graph,

where $\circ$ stands for the element-wise dot. Following Eq. (6), we conduct graph attention operation based on $\mathcal{G}_{\rm intra}^{S}$.

We compare the proposed FGRR with state-of-the-art DAOD methods. (1) Domain alignment: Domain adaptive Faster-RCNN (DA-Faster) [16], Strong-Weak Distribution Alignment (SWDA) [18], Domain Diversification and Multi-domain-invariant Representation Learning (DD-MRL) [100], Collaborative Training (CT) [72], Augmented Feature Alignment Network (AFAN) [78], CDTD [99], and Scale-Aware DA-Faster (SA-DA-Faster) [79]. (2) Local alignment: Mean Teacher with Object Relations (MTOR) [19], Selective Cross-Domain Alignment (SCDA) [17], Multi-Adversarial Faster-RCNN (MAF) [20], Conditional Domain Normalization (CDN) [71], Progressive Disentanglement (PD) [26], Image-level Categorical Regularization and Categorical Consistency Regularization (ICR-CCR) [24], Coarse-to-Fine (C2F) [23], Asymmetric Tri-way Faster-RCNN (ATF) [77], Graph-induced Prototype Alignment (GPA) [22], Hierarchical Transferability Calibration Network (HTCN) [21], Prior-Adversarial Loss (PAL) [73], Memory Guided Attention for Category-Aware Domain Adaptation (MeGA-CDA) [74], Vector-Decomposed Disentanglement (VDD) [76], and Dual Bipartite Graph Learning (DBGL) [30].

For all the aforementioned methods, we cite the quantitative results from their original papers. Source Only stands for the model that is trained only using source images and directly evaluated on the target domain without any adaptation. To facilitate a fair comparison, we utilize the gain of performance compared to the Source Only model as the main evaluation metric by following [23].

Real-to-Artistic. Table I, Table III, and Table IV report the adaptation results of Pascal VOC $\rightarrow$ Clipart, Pascal VOC $\rightarrow$ Watercolor, and Pascal VOC $\rightarrow$ Comic. The proposed FGRR significantly outperforms all comparison methods on different real-to-artistic adaptation tasks. In particular, compared to the preliminary version (DBGL), the proposed FGRR improves over its results by +2.2% on average (41.6% $\rightarrow$ 43.3%, 53.8% $\rightarrow$ 55.7%, and 29.7% $\rightarrow$ 32.7%), which clearly demonstrates the significance of our improvements. In contrast to adversarial-based domain alignment (e.g., DA-Faster and SWDA) and local alignment approaches (e.g., MAF, HTCN, MeGA-CDA, and VDD), FGRR additionally explore the topological interactions on both intra- and inter-domains to enable fine-grained knowledge transfer. Our better performance over them can explicitly testify the effectiveness of these two components. ATF resorts to the tri-training strategy to solve the labeling unfairness problem between source and target domains, and thus is complementary to our FGRR. That is to say, our work focuses on adapting internal features without requiring model ensemble (e.g., mean-teacher [19] and tri-training [77]). The better performance of FGRR further verifies the importance of endowing DAOD model with the relational reasoning ability. In addition, since the proposed model explicitly consider the imbalance problem in DAOD, the per-class adaptation performance is more balanced. For instance, in Table IV, the performances of “bird”, “car” “cat”, and “dog” are balanced, and their average performance is substantially improved at the same time.

Normal-to-Foggy. Table II displays the results on adaptation from Cityscapes to Foggy-Cityscapes. This scenario is the most widely used DAOD datasets, and contains distinct object co-occurrence, backgrounds, and weather. The source and target images are originally the same one, which makes this task more suitable for explicit local feature alignment approaches, such as HTCN [21] and MeGA-CDA [74]. As can be seen, the proposed FGRR achieves a remarkable increase of +20.5% over the Source Only model on adaptation between these two similar domains, verifying the robustness and effectiveness of FGRR on the different DAOD scenario. The results also reveal several interesting observations.

1. 1.

   The series of holistic feature alignment approaches (such as DA-Faster, SWDA, and DD-MRL) demonstrate inferior performance since they do not consider the local nature of object detection problem. In other words, the objects of interest only take a small portion of the whole image, and thus globally aligning features between domains is prone to result in negative transfer.

2. 2.

   Local alignment approaches (such as SCDA, C2F, MeGA-CDA, HTCN, and VDD) change the focus of the adaptation from holistic to local by virtue of elaborate local alignment methods regarding the foreground objects. In this way, they substantially improve the adaptation performance compared to holistic alignment approaches.

3. 3.

   MTOR explores the object relation within and between domains in the context of a mean-teacher framework. However, they focus on optimizing consistency regularization without delving into the interactions and dependencies of different foreground objects. By contrast, the proposed FGRR utilizes the bipartite graph to model the cross-domain relations where two set of features are aggregated over the spatial and semantic spaces. Noting that FGRR significantly outperforms the result of MTOR by 5.7% (from 35.1% to 40.8%).

4. 4.

   C2F and GPA aim to achieve the cross-domain semantic consistency based on prototype alignment. In particular, GPA utilizes the graph-based information propagation to obtain the prototype representation of each category. However, these two approaches heavily rely on the region proposal step to generate instance-level features, which may be noisy due to the error accumulation problem. More importantly, they neglect the valuable low-level features and the rich interactions among objects of different categories. Our better performance over them (from 38.6% and 39.5% to 40.8%) further reveals the importance of relational reasoning for DAOD tasks.

5. 5.

   In essence, the classical one-vs-one alignment (such as adversarial training) can be seen as the simplest case of relational reasoning. Most conventional domain adaptation methods assume that perfect alignment equals to precise knowledge transfer, while the many-vs-many relations between different entities are ignored. Moreover, alignment-based approaches naturally neglect the intra-domain relations as no entities can be explicitly aligned within each domain. In this regard, our work gives a hint to bridge the gap between alignment-based and relational reasoning based DAOD.

Synthetic-to-Real. Table V presents the results on adaptation from Sim10K to Cityscapes. From the table, we can observe that the proposed FGRR algorithm significantly improves the baseline methods. The domain disparity of this adaptation task mainly stems from the difference of image style, making it more sophisticated than normal-to-foggy adaptation where only weather changes. In that case, we argue that FGRR explicitly extract the source and target foreground pixels and regions to promote the positive interactions of foregrounds via relational reasoning procedure as well as mitigate the negative influence of background noises. By comparison, prior efforts, which aim at either holistic image alignment or local region alignment, cannot ensure the precise knowledge transfer among foreground objects across domains. In particular, methods introducing a FPN architecture would significantly improve over the baseline models. The justification is that this adaptation task focuses on detecting cars which have large scale variances across different scenes, naturally fitting the feature pyramid property of FPN architecture. However, most existing works do not integrate the FPN into their detection backbone, and it may be non-trivial to combine them especially when the DAOD methods have adaptation modules within the backbone networks. To facilitate a fair comparison, we follow the mainstream settings.

Cross-Site Mammogram Mass Detection. Table VI provides the experimental results on mammogram mass detection, which targets on testing the scalability and efficacy of the proposed FGRR on real-world applications. Cross-site mammogram mass detection substantially differs from adaptation based on nature images due to the particular challenges of sparse distribution and tiny size of lesions, difference of imaging quality as well as existence of hard mimics. According to the table, we can see that the proposed FGRR outperforms all comparison methods by a large margin on this challenging adaptation task (from 38.4% to 43.9%), clearly demonstrating the effect of our approach on highlighting foreground pixels versus regions and explicitly modeling foreground interactions on both intra- and inter-domains. Moreover, such practical application reveal the potential of DAOD model for improving complex problems.

We investigate the individual effects of each proposed module by performing in-depth and complete ablation studies. We evaluate several variants of FGRR: (1) FGRR w/o PRR, FGRR w/o SRR, and FGRR w/o IOR: remove the pixel-level and semantic relational reasoning modules as well as the image-level object-aware reweighting module from the full FGRR model respectively. (2) PRR w/o inter and PRR w/o intra: remove the inter- and intra-domain reasoning modules from the PRR respectively. SRR w/o inter and SRR w/o intra: remove the inter- and intra-domain reasoning modules from the SRR respectively. (3) PRR w/o BGL: directly learn the cross-domain one-vs-one correspondence based on searched foreground pixel pairs without introducing bipartite graph. PRR w/ random link: randomly select pixels to construct bipartite graph. (4) SRR w/o BGL: directly learn the cross-domain one-vs-one correspondence based on foreground proposals without introducing bipartite graph.. SRR w/o CDA: remove the category-aware domain alignment module from SRR. The results of ablation study are summarized in Table VIII. From the table, we have the following observations. (1) Respectively removing the PRR and SRR components from the full FGRR model will lead to substantial performance degeneration, clearly revealing their individual effects and the complementary effect between intra- and inter-domain relational reasoning. (2) The results of FGRR w/o IOR verify that estimating the object-level variations based on image-level features can harmonize the adaptation process and thus provide better knowledge transfer. (3) Inter-domain and intra-domain relational reasoning modules make almost equal contributions to the final performance, highlighting the necessity of considering them in a unified framework. (4) The performances of PRR w/o BGL and SRR w/o BGL drop significantly compared to the full FGRR model, demonstrating the importance of performing relational reasoning compared to conventional local and global alignment approaches. (5) The results of PRR w/ random link reveal that randomly linking graph node will bring in a number of noisy connections and thereby cause significant performance degeneration, showing the superiority of graph construction strategies in our method.

Considering that the IoU threshold is an important factor that affects the detection results, we study the adaptation performance of three models (i.e., Source Only, HTCN [21], and FGRR) with different IoU threshold on Cityscapes $\rightarrow$ Foggy-Cityscapes and Pascal VOC $\rightarrow$ Clipart1k. The results are reported in Figure 3, where we can see that 1) the mAP gradually decreases with the increasing of IoU threshold and approaches zero at the end, and 2) the proposed FGRR consistently outperforms the comparison models on different threshold, revealing the effectiveness of our FGRR on providing robust and precise bounding boxes prediction.

Figure 4 visualizes the image-level features generated by Source Only, HTCN and our FGRR using t-SNE algorithm on adaptation task Pascal VOC $\rightarrow$ Clipart1k. As can be seen, for our FGRR model, images that have similar semantic information will get much closer in the feature space compared to the other models. It is noteworthy that when compared to Source Only and HTCN, FGRR does not match the source and target features more compact, but improves the adaptation performance by a large margin in terms of mAP, implying that globally matching features cannot ensure the fine-grained knowledge transfer. This result also verifies the hypothesis in [18] that weak global alignment helps the adaptation of object detectors.

Figure 5, Figure 6, and Figure 7 provide some detection results on the target domain based on different DAOD methods, i.e., Source Only, HTCN [21], and FGRR. From the figures, we can see that the proposed FGRR is significantly better than the compared methods in terms of giving accurate boxes regression and object classification. More specifically, FGRR is capable of precisely detecting those sample-scarce or/and hard-to-classify categories, obscured or/and tiny foreground objects, as well as largely reducing the false positive results.