Unsupervised Domain Adaptation for One-stage Object Detector using Offsets to Bounding Box

Deep-learning-based object detection can be categorized into anchor-based and anchor-free methods. Anchor-based detectors define various sizes and ratios of anchors in advance and utilize them to match the output of the detector with the ground-truth. On the other hand, anchor-free detectors do not utilize any anchors but rather directly localize objects employing fully convolutional layers. Moreover, depending on whether region proposal network (RPN) is used or not, object detectors can also be classified into two-stage and one-stage detectors. Faster R-CNN [29] is a representative anchor-based two-stage detector while SSD [22] and YOLO [28] are anchor-based one-stage detectors. There are some renowned anchor-free one-stage detectors as well. Cornernet [20] and Centernet [10] localize objects by predicting the keypoints or the center of an object while FCOS [36] directly computes the offset from each location on the feature map to the ground-truth bounding box. Most works of DAOD have been conducted on Faster R-CNN, a two-stage detector and relatively few works have been done using an anchor-free one-stage detector. There are several works [3, 30, 18] that have been conducted on SSD, a representative one-stage anchor-based detector and only [14, 25] carried out domain adaptation using FCOS, an anchor-free one-stage detector. Our work focuses on boosting the domain adaptation performance on FCOS [36] leveraging its anchor-free architecture and fast speed.

There are three main approaches of UDA for object detection tasks: adversarial alignment, image translation, and self-training. Image-translation-based methods translate the source domain images into another domain using a generative model [19, 9, 34, 2, 30, 15] to adapt to the target domain. Self-training-based methods [31, 17, 9, 18, 25] generate pseudo-labels for the target domain images with the model pre-trained on the source domain and re-train the model with the pseudo labels. For adversarial alignment methods, [5] is a seminal work that aligns the feature distribution of the source and the target domain using a domain discriminator based on the Faster R-CNN [29]. Since then, there have been studies to align feature distribution at multiple levels [32, 13], studies focusing on the importance of local features in detection, and studies to align instance-level features that may correspond to objects [39, 44, 14]. Recently, there have been attempts [3, 37, 45, 40, 46] to align the instance-level features in a class-wise manner, focusing on the fact that the distribution of instance-level features is clustered by class. Based on our observation that the feature distribution varies depending on the offset values in FCOS [36] and the detection task requires not only classification but also regression, we propose an adversarial alignment scheme with state-of-the-art performances in various experimental settings by aligning the features in an offset-aware manner.

FCOS [36] is a representative one-stage detector that predicts object categories and bounding boxes densely in feature maps without a RPN. FCOS uses five levels of feature maps ($F_{3}\sim F_{7}$) produced from the backbone network following FPN [21] to detect various sizes of objects. At each location of the feature map, it predicts the corresponding object category, the centerness indicating how central the current location is to the object, and the distances from the current location to the left, top, right, and bottom $(l,t,r,b)$ of the nearest ground-truth bounding box as shown in Fig. 3. With a design that five levels of feature maps have different resolutions decreasing by a factor of $1/2$, the maximum distance responsible for each feature level is set to ($64,128,256,512,\infty$).

FCOS differs from conventional anchor-based or RPN-based object detectors which regress four values to correct anchors or proposals. We observe that the features are distributed according not only to object categories but also to offsets, the distances to the nearest bounding boxes, and it is more pronounced in FCOS due to its characteristics of predicting $(l,t,r,b)$ directly using the features. Focusing on this observation, our proposed method tries to align the features of the source and the target domains by conditioning the features with the offsets.

Consider the setting where we adapt the object detector to the target domain using labeled source domain data $\mathbb{D}_{S}$ and unlabeled target domain data $\mathbb{D}_{T}$ which share the same label space consisting of $C$ classes. When training a detector, we only have access to $\mathbb{D}_{S}={(x_{i}^{S},y_{i}^{S},b_{i}^{S})}_{i=1}^{N_{S}}$ and $\mathbb{D}_{T}={(x_{i}^{T})}_{i=1}^{N_{T}}$, where $x_{i}$ is the input image, and $y_{i}\in[C]^{k\times 1}$ and $b_{i}\in\mathbb{R}^{k\times 4}$ are the object categories and the bounding box coordinates of all the $k$ objects existing in $x_{i}$. $N_{S}$ and $N_{T}$ are the numbers of samples in $\mathbb{D}_{S}$ and $\mathbb{D}_{T}$.

To ensure that the object detector works well on the target domain, we align the features of both the source and the target domain to have a marginally similar distribution through an adversarial aligning method using a global domain discriminator $D_{g}$. Let $F^{S}$ and $F^{T}$ be the feature maps obtained by feeding the source domain image $x^{S}$ and the target domain image $x^{T}$ to the backbone, respectively. When the spatial size of a feature map $F$ is $H\times W$, $D_{g}$ is trained to classify the domain of the feature map $F$ pixel-wisely by the binary cross entropy loss as in (1). The label of the source domain is 1, while that of the target is 0.

Using the gradient reversal layer (GRL) proposed in [11], the backbone is adversarially trained to prevent the domain discriminator from correctly distinguishing the source and the target domains, thereby generating domain-invariant features.

In order to align features in an offset-aware manner, the features and the corresponding offsets can be concatenated and inputted to the domain discriminator. However, simply concatenating them is not enough to fully utilize the correlation between features and offset values. Inspired by [24], which considers the correlation between features and categorical predictions, our method does not simply concatenate but outer-product features and offset values. Unlike classification which uses a categorical vector, an offset value is a continuous real value. Hence, the product of the feature and the offset value only has the effect of scaling the feature. To effectively condition the feature according to the corresponding offset values, each offset value of $(l,t,r,b)$ is converted into a probability vector corresponding to $N_{bin}$ bins using (2). In the equation, $z_{u,v}^{n}$ refers to the offset value for $n\in\{l,t,r,b\}$ at the location $(u,v)$ in the feature map. To convert the offset $z_{u,v}$ to an $N_{bin}$-dimensional probability vector, we calculate the probability using the distance of the log of the offset value to a predefined value $m_{i}$ for the $i$-th bin. Note that we have $N_{bin}$ bins and $m_{1}<m_{2}<\cdots<m_{N_{bin}}$. Assuming that the probability of $z_{u,v}$ belonging to the $i$-th bin is proportional to a normal distribution with its mean $m_{i}$ and a shared variance $\sigma^{2}$, it becomes as follows:

Here, $\tau$ is a temperature value to make the distribution smooth, and in all of our experiments, both $\sigma$ and $\tau$ are set to 0.1. In all of our main experiments, $N_{bin}$ is set to 3 for each feature level, and $m_{i}$ for each bin is set to $(m_{1},m_{2},m_{3})=(lv-\frac{1}{2},lv+\frac{1}{2},lv+\frac{3}{2})$ for $lv$-th level ($lv\in\{3,...,7\}$) to satisfy $\frac{m_{1}+m_{2}}{2}=lv$ and $\frac{m_{2}+m_{3}}{2}=lv+1$ because each feature level is responsible for a different object scale. Fig. 4 shows the probability of $z_{u,v}$ belonging to each bin when there are three bins. The obtained probability vector ${q}\triangleq[q_{1},\cdots,q_{N_{bin}}]^{T}\in\mathbb{R}_{+}^{N_{bin}}$ still maintains the relative distance relationship of the real offset value as $D_{KL}({q}(a)||{q}(b))<D_{KL}({q}(a)||{q}(c))\ \text{if}\ \ a<b<c$.

However, the probability vector is uniformly initialized for all ${N_{bin}}$ bins since the regressed offsets may not be accurate at the beginning of training. During the first $I$ warm-up iterations, we gradually increase the rate of using the probability vector ${q}_{u,v}$ as iteration ($iter$) progresses utilizing the alpha-blending as follows:

Here, $\alpha_{0}$ is the constant value between 0 and 1 that smoothes the probability vector $\tilde{q}$, and the closer it is to 1, the more uniform $\tilde{q}$ becomes. We show the effects of $\alpha_{0}$ in Sec.4.5. In all the experiments, $I$ is set to 6k, the half of the first learning rate decay point. $\mathbbm{1}$ is a vector consisting of only ones.

Using $\tilde{{q}}_{u,v}$, we can obtain the features which are conditional to the offset values by outer-producting them as follows:

Here, $f_{u,v}\in\mathbb{R}^{D}$ is a feature vector located at location $(u,v)$ in the feature map $F\in\mathbb{R}^{D\times HW}$ and $\tilde{q}_{u,v}\in\mathbb{R}^{N_{bin}}_{+}$ is the probability vector of the corresponding offset value obtained from (3). By outer-producting $f_{u,v}$ and $\tilde{q}_{u,v}$, we can get a new feature, $g_{u,v}\in\mathbb{R}^{D\times N_{bin}}$ conditioned on the offsets. By flattening $g_{u,v}$, the conditioned feature map $G\in\mathbb{R}^{(D\times N_{bin})\times HW}$ with the same spatial resolution as $F$ is obtained, which is fed into the domain discriminator $D_{c}$.

Outer product is effective in conditioning because it considers the correlation between features and offsets without loss of information, hence it enables features to have different characteristics depending on offset values. Consider a case where $N_{bin}=3$ and $(m_{1},m_{2},m_{3})=(3.5,4.5,5.5)$ for $F_{4}$, and conditionally align to $t$, the top offsets. Suppose that feature vector $f_{u_{1},v_{1}}\in\mathbb{R}^{D}$ located at $(u_{1},v_{1})$ have a small top offset prediction, i.e. $t=13$ and $\log_{2}t=3.7$, resulting in the probability vector $q_{u_{1},v_{1}}=(0.98,0.02,0.0)$ via (2). Conditioned feature $g_{u_{1},v_{1}}$ obtained by outer-producting $f_{u_{1},v_{1}}$ and $q_{u_{1},v_{1}}$ is a $3\times D$ matrix. The first row of $g_{u_{1},v_{1}}$ would be similar to the original feature $f_{u_{1},v_{1}}$, but the elements in the other rows would be almost zero. On the other hand, $g_{u_{2},v_{2}}$ obtained by e.g. $q_{u_{2},v_{2}}=(0,0.01,0.99)$ with a large top offset would have the original feature $f_{u_{2},v_{2}}$ in the third row but have almost zero elements in another rows. Therefore, a domain discriminator is trained to classify the domain of the features in different subspaces according to the offsets. As a result, the backbone will generate features that are domain invariant conditioned on offsets to fool the discriminator.

To generate a feature conditioned on the offsets, we need to know which location in the feature map corresponds to the object and the accurate offset value at that location. For the labeled source domain, we can easily obtain the ground-truth value of which location corresponds to the object in each feature map and the offsets. Since FCOS calculates the object mask corresponding to each category in the feature map and the distance from each location to the GT bounding box, we can utilize this mask and the ground-truth offset values, $(l^{*},t^{*},r^{*},b^{*})$. On the other hand, in the unlabeled target domain, we should inevitably use predicted values. Although classification confidence is the probability of predicting the category of an object rather than a regression, by using it, we can simply and effectively select instance-level features with high objectness and obtain reliable regression values for those features. Fig. 5 is the analysis of features obtained by feeding a target domain image into the globally aligned model of Sec. 3.3. The blue bar represents the ratio corresponding to the actual objects among the features having a confidence value higher than the threshold value of the x-axis, and the red line represents the average of the difference between the regression values and the GT offsets of the features. The higher the confidence threshold, the higher the probability of the feature’s location belonging to an actual object, and the closer the regression value is to the GT offset value. In $p_{cls}^{u,v}\in\mathbb{R}_{+}^{C}$, which is the category classification probability at the $(u,v)$-th location of the feature map, $\max_{c\in[C]}(p_{cls}^{u,v})$ (the maximum probability among all classes) can be viewed as objectness which is the probability that the location corresponds to an object. Therefore, we generate a mask where $\max_{c\in[C]}(p_{cls}^{u,v})$ is higher than a threshold $\rho$, and we weight the activated part with that max probability value as follows:

Finally, we align the distribution of features $G$ of the source and the target which is conditioned on the offset by minimizing the following adversarial loss:

Here, $\mathcal{L}_{c}^{n}$ is the adversarial loss conditional to the offset value $n\in\{l,t\}$ using the discriminator $D_{c}^{n}$. Since the correlation between the regression values for the left $l$ and the right $r$ and between the top $t$ and the bottom $b$ are strong, conditioning is performed only for the left and the top. $d$ represents whether the domain is the source or the target. $\mathbbm{I}$ is the indicator function and $\odot$ is the elementwise multiplication. Through this, the instance-level features of both domains can be aligned to have the same distribution conditional to the offset.

Using labeled source domain data, the backbone and heads of FCOS are trained by minimizing object detection loss $\mathcal{L}_{det}$ consisting of object classification loss $\mathcal{L}_{det\mbox{-}cls}$ and bounding box regression loss $\mathcal{L}_{det\mbox{-}reg}$, as in [36]:

In addition, we introduce $\mathcal{L}_{g}$ in (1) to ensure that the overall features of both domains have the similar marginal distribution and $\mathcal{L}_{c}$ in (6) to allow the instance-level features to have the same conditional distribution to offsets, as follows:

where $\lambda_{g}$ and $\lambda_{c}$ are parameters balancing the loss components.

We conduct experiments on three scenarios: adaptation to adverse weather driving (Cityscapes to Foggy Cityscapes, i.e. CS $\rightarrow$ FoggyCS), adaptation from synthetic data to real data (Sim10k to Cityscapes, i.e. Sim10k $\rightarrow$ CS), and adaptation to a different camera modality (KITTI to Cityscapes, i.e. KITTI $\rightarrow$ CS).

• •

  Cityscapes [8] consists of clear city images under driving scenarios, summing to 2,975 and 500 images for training and validation, respectively. There are 8 categories, i.e., person, rider, car, truck, bus, train, motorcycle and bicycle.

• •

  Foggy Cityscapes [33] is a synthetic dataset that is rendered by adding fog to the Cityscapes images. We use Cityscapes as the source, and Foggy Cityscapes as the target to simulate domain shift caused by the weather condition.

• •

  Sim10k [16] consists of 10,000 synthesized city images. For the adaptation scenario from synthetic data to real data, we set Sim10k as the source domain and Cityscapes as the target domain. Only car class is considered.

• •

  KITTI [12] consisting of 7,481 images is a driving scenario dataset similar to Cityscapes, but there is a difference in camera modality. For adaptive scenarios to other camera modalities, we use KITTI as the source and Cityscapes as the target. Similar to Sim10k to Cityscapes, only car category is used.

We use VGG-16 [35] backbone and fully-convolutional head consisting of three branches of classification, regression and centerness following [36]. For the domain discriminators, $D_{g}$ and $D_{c}$, fully-convolutional layers with the same structure as the head are used. We initialize the backbone with the Image-Net pretrained model and reduce the overall domain gap using only object detection loss $\mathcal{L}_{det}$ and global alignment $\mathcal{L}_{g}$ at the beginning of training. Then, we train the model for 20k iterations with weight decay of 1e-4, initial learning rate of 0.02 for CS $\rightarrow$ FoggyCS, 0.01 for Sim10k $\rightarrow$ CS and 0.005 for KITTI $\rightarrow$ CS, respectively. We decay the learning rate at 12k and 18k iteration by the rate of one-tenth. During training, $\lambda_{g}$ and $\lambda_{c}$ are fixed as 0.01 and 0.1, respectively. We set the weight for the Gradient Reversal Layer (GRL) to 0.02 for global alignment and 0.2 for our conditional alignment. Also, we set the confidence threshold in (5) as $\rho=0.3$ for CS $\rightarrow$ FoggyCS and $\rho=0.5$ for Sim10k, KITTI $\rightarrow$ Cityscapes to reduce the effects of incorrect predictions. We set $I$ to 6k which is the half of the first learning rate decay point and $\alpha_{0}$ in (3) to 0.2. Input image is resized to 800 for shorter side, and 1333 or less for longer side following [14, 25, 36].

In Table 1, we compare the performance of our method (OADA) with other existing methods in CS $\rightarrow$ FoggyCS setting. When EPM [14] based on FCOS is trained in the exact same setting as ours, the performance is 3.9%p higher than what was reported. Conditional alignment on the offsets to the left and the top side of the bounding box improves performance by 22.2%p and 22.3%p respectively compared to Source Only, by 5.7%p and 6.0%p compared to GA^† which is only globally aligned and by 2.7%p and 2.8%p compared to EPM* that we reimplemented. Since Foggy Cityscapes has 8 categories of objects and has various aspect ratios, when conditioning is performed on both the left and the top offsets, the additional performance gain is very larger by 2.1%p or more, achieving the state-of-the-art regardless of the detector architecture. By initializing with the OADA (Offset-Left & Top) pre-trained model and training once more with self-training [23], we get a model almost similar to Oracle lagging only by 0.2%p. Detailed implementation of self-training is explained in the supplementary.

Table 2 shows the adaptation results of Sim10k and KITTI $\rightarrow$ CS. In Sim10k $\rightarrow$ CS, conditional alignment using left and top offsets improves mAP by 15.0%p and 15.3%p, respectively, compared to Source-Only and 4.3%p and 4.6%p over re-implemented EPM*. Likewise, in KITTI $\rightarrow$ CS, our methods using left and top offsets improve the performance by 1.4%p and 1.6%p, respectively, over Source-Only and 1.9%p and 2.1%p over EPM*. In these two benchmarks, only the car class is considered, so the gain of OADA (Offset-Left & Top) is not as large as the multi-category setting, but there are still additional gain of 0.9%p and 0.5%p for Sim10k and KITTI. Our conditional aligning alone already achieves the state-of-the-art performance but greater performance can be obtained by employing the self-training as in OADA (Offset-Left & Top + Self-Training).

Chen et. al [4] argued that aligning the feature distribution through adversarial alignment increases the transferability of features, but they did not take the discriminability into account, the ability to perform tasks well. They measure the discriminability via singular value decomposition (SVD) of the target domain feature maps and measure the transferability by estimating the corresponding angle between the feature spaces of the source and the target domain. Using their proposed metrics [4, 42], we compare the discriminability and transferability of Ours (OADA) with Source Only, GA of Sec. 3.3, EPM [14] and Oracle models in CS$\rightarrow$Foggy CS. When $F^{S}=[f_{1}^{S}...f_{N_{S}}^{S}]$ and $F^{T}=[f_{1}^{T}...f_{N_{T}}^{T}]$ are feature matrix of the source and the target domain respectively, we apply SVD as follows:

$\alpha_{0}$ in (3): Table 3 shows the effect of $\alpha_{0}$ that smoothes $\tilde{q}$ used for feature conditioning. When $\alpha_{0}=0$, conditioning is done using only $q$, which is the probability vector of an offset, without smoothing after the warm-up period. Since the softmax in (2) makes $q$ almost one-hot vector in most cases, the values of certain rows of conditioned feature $g$ are almost zero. This strongly constrains the dimension of the conditioned features. On the other hand, when $\alpha_{0}=1$, we outer-product the feature with an uniform vector $\frac{1}{N_{bin}}\mathbbm{1}$ without offeset-aware conditioning throughout training. In this case, the feature dimension given to the discriminator is the same, but performance is greatly degraded because there is no conditioning according to the offset. $\alpha_{0}=0.2$, in which relatively strongly constrained features are used in the discriminator, is most appropriate.