EARL: An Elliptical Distribution aided Adaptive Rotation Label Assignment for Oriented Object Detection in Remote Sensing Images

Due to its wide range of application scenarios, oriented object detection in RSIs has developed rapidly. Similar to generic detectors, recent methods can be divided into two categories, i.e., anchor-based detectors and anchor-free detectors. To detect targets with arbitrary orientations, some anchor-based detectors[33, 34, 35], which are built on the two-stage framework, i.e., Faster R-CNN[8], densely preset multiple anchors with different scales, aspect ratios and angles for better regression while introducing heavy anchor-related computations. After that, RoI-Transformer[36] proposed an efficient region of interests (RoI) learner to transform horizontal proposals to rotated ones. Gliding Vertex[37] described oriented targets by horizontal bounding boxes (HBB), which further avoided numerous anchors with multiple angles. Then, DODet[38] was proposed to deal with the spatial misalignment between horizontal proposals and oriented targets, as well as the feature misalignment between classification and localization. Whereas R³Det[21] and S²A-Net[20] followed the one-stage schema to balance accuracy and efficiency, and alleviated the feature misalignment issue. However, these methods still take a long time during training and inference[19].

To maintain high efficiency, anchor-free detectors are designed to avoid the usage of anchors. For instance, TOSO[30] directly regressed the surrounding HBB and transformation parameters to present rotated targets. IENet[17] was built upon the classic one-stage anchor-free framework, i.e., FCOS, adding an additional branch with the interactive embranchment module to regress angles. Later, BBAVector[39], O²-DNet[40] and ACE[14] employed a simplified architecture and designed new oriented bounding box (OBB) representation methods. However, these anchor-free detectors still not perform the state-of-the-art results compared with anchor-based detectors, and most of them are more likely to improve the architecture while not considering the training strategy, especially the label assignment strategy, which has been proven to be the essential difference between these two kinds of detectors[27].

Label assignment is a crucial component for convolutional neural network (CNN) based detectors to learn feature distribution of targets by determining positive or negative samples. Meanwhile, as the current detectors commonly adopt multi-scale feature maps to alleviate the scale variations, label assignment needs to simultaneously assign labels to samples at different spatial locations (spatial assignment) on different levels of feature maps (scale assignment)[26]. Most anchor-based detectors[8, 10, 18] assign the labels to anchors among multi-level feature maps by comparing the preset thresholds and IoU between anchors and ground truth bounding boxes. However, this strategy often involves many hyperparameters to adjust the scales and aspect ratios of anchors based on the datasets[41]. Whereas anchor-free methods[11, 42, 43] often select samples inside the ground truth bounding box or the central area of targets for spatial assignment, and choose samples on a specific level of feature maps according to the predefined target size threshold for scale assignment. However, above strategies, which depend on the heuristic rules, may not be optimal enough.

Thus, recent works provide dynamic label assignment strategies, which allow models to learn to select samples by itself. For instance, FSAF[23] dynamically assigned targets to the suitable feature levels based on computed loss. ATSS[27] proposed an adaptive training sample selection method by adjusting the IoU threshold according to the statistics of targets, which also proposed the center sampling strategy to improve the quality of positive samples. Autoassign[26] used two weighting modules to adjust the category-specific prior distribution according to the appearances of targets. As for oriented object detection, SASM[44] was motivated by the idea of ATSS and designed a dynamic IoU threshold that incorporates shape information. RTMDet[32] proposed a dynamic soft label assignment based on SimOTA[45]. GGHL[31] used a Gaussian heatmaps to define positive samples according to the shape and direction properties of targets. AOPG[15] applied a region assignment scheme to select positive samples within the shrinked OBB area. Oriented RepPoints[46] used an adaptive quality assessment and sample assignment scheme. FSDet[19] assigned a soft weight to each sample among all levels of feature maps. However, the label assignments used by these methods are rarely consider the specific characteristics of targets in RSIs, i.e., large variations in scales and aspect ratios, resulting in insufficient and imbalanced sampling and often introducing large amounts of low-quality samples. Therefore, this study focuses on designing an adaptive label assignment strategy for orientation anchor-free detectors by taking into account the important characteristics in RSIs to address aforementioned problems.

To demonstrate the effectiveness of the proposed label assignment strategy, in this paper, a simple but classical anchor-free architecture, namely R-FCOS, is employed as our baseline detector. Compared with FCOS[11], R-FCOS removes centerness branch to be more compact and achieves more simplicity and efficiency, which only contains classification branch and regression branch for oriented object detection.

As shown in Fig. 3, the upper gray block shows the baseline architecture, which consists of the backbone network, feature pyramid network (FPN) and prediction heads. Let $\mathbf{C}_{\ell}$ and $\mathbf{P}_{\ell}\in\mathbb{R}^{H_{\ell}\times W_{\ell}\times\mathcal{C}_{\ell}}$ be the feature maps from backbone network and FPN, respectively, where $\ell$ is the layer index, $H_{\ell}\times W_{\ell}$ represents the size of feature maps and $\mathcal{C}_{\ell}$ denotes the number of feature channels. In this work, following FCOS, five levels of multi-scale feature maps $\{\mathbf{P}_{3},\mathbf{P}_{4},\mathbf{P}_{5},\mathbf{P}_{6},\mathbf{P}_{7}\}$ are used, where $\mathbf{P}_{3},\mathbf{P}_{4}$ and $\mathbf{P}_{5}$ are generated from $\mathbf{C}_{3},\mathbf{C}_{4}$ and $\mathbf{C}_{5}$, respectively. Whereas $\mathbf{P}_{6}$ and $\mathbf{P}_{7}$ are generated via up-sampling $\mathbf{P}_{5}$ and $\mathbf{P}_{6}$, respectively.

After obtaining the feature maps from FPN, the prediction heads with two fully convolutional subnets are employed to predict the categories and regressions for each location on feature maps. Specifically, given a set of ground truths $\mathcal{G}=\{g_{i}\}_{i=1}^{n}$ in the input image, where $n$ denotes the total number of ground truths and $i$ is the index, each $g_{i}$ in $\mathcal{G}$ is represented by $g_{i}=(x_{i},y_{i},w_{i},h_{i},\theta_{i},c_{i})$, as shown in Fig.  4, where $(x_{i},y_{i})$ is the coordinate of center point of $g_{i}$, $w_{i}$ and $h_{i}$ represent the width and height of $g_{i}$, and $\theta_{i}\in[0^{\circ},180^{\circ})$ represents the counterclockwise angle between the y-axis and long side, and $c_{i}$ is the class label that the ground truth belongs to.

To generate prediction, we adopt $\mathcal{A}_{\ell}=\{\alpha^{\ell}_{j}\}_{j=1}^{H_{\ell}\times W_{\ell}}$ to denote the set of anchor points, which are the pixel-wise locations on $\mathbf{P}_{\ell}$. For each anchor point $\alpha^{\ell}_{j}$, it can be mapped back to the input image via:

where $(\tilde{x}^{\ell}_{j},\tilde{y}^{\ell}_{j})$ and $(x_{j},y_{j})$ represent the locations of $\alpha_{j}^{\ell}$ on $\mathbf{P}_{\ell}$ and the input image, respectively. $s_{\ell}$ denotes the stride on $\mathbf{P}_{\ell}$ and $\lfloor\cdot\rfloor$ denotes the round-down operator. Here, we directly regress the target bounding box at each $\alpha_{j}^{\ell}$. Note that, the label assignment strategies are often employed to decide whether each $\alpha_{j}^{\ell}$ is a positive or negative sample, and allow the network to select higher-quality samples for training.

Then, if $\alpha_{j}^{\ell}$ is a positive sample and assigned to the ground truth, e.g., $g_{i}$, its class label $c^{*}_{j}$ equals to $c_{i}$ and it has a 5-dimensional vector $t_{j}^{*}=(x^{*}_{j},y^{*}_{j},w^{*}_{j},h^{*}_{j},\theta^{*}_{j})$ being the regression targets for the location, where $x^{*}_{j}$ and $y^{*}_{j}$ are the coordinate offsets between the center point of $g_{i}$ and $\alpha_{j}^{\ell}$, as shown in Fig. 4, which can be formulated as:

$w^{*}_{j}$, $h^{*}_{j}$ and $\theta^{*}_{j}$ equal to the width, height and angle of $g_{i}$, respectively.

During inference, the prediction heads will directly generate classification and regression for each anchor point and output the classification scores and locations after post-processing, hence our proposed strategy is inference cost-free.

The above represents the pipeline of the simple baseline detector, as illustrated in Fig. 3. This simple baseline detector is only used for illustration to show how our proposed EARL strategy works on the baseline detector for sample selection by simply replacing its original label assignment strategy.

Note that, the proposed EARL strategy can be also easily deployed on other baseline detector in the same way. Specifically, we also equipped it with an advanced anchor-free detector, i.e., RTMDet-R [32], for performance evaluation to demonstrate the effectiveness of our study in the experiments.

In the next subsections, the components of our proposed EARL, which are given in the lower green block of Fig. 3, will be introduced in detail.

Different from the existing scale assignment strategies, the simple but novel ADS strategy is proposed to select positive samples adaptively according to the scales of targets. Its implementation can be summarized as sampling sequentially from high to low-level feature map, e.g., from $\mathbf{P}_{7}$ to $\mathbf{P}_{3}$, as illustrated in Fig. 3.

Specifically, for each $g_{i}$ in the input image, a determined function $\tau(\cdot)$ is used to indicate whether each $\alpha^{\ell}_{j}$ is inside a certain region of $g_{i}$ when mapped back to the input image, which can be simply understood as selecting samples inside the mask region on each $\mathbf{P}_{\ell}$, as shown in Fig. 3. Then, if $\alpha^{\ell}_{j}$ is judged to be inside the mask region by $\tau(\cdot)$, it will be selected into the candidate samples set $\mathcal{R}_{\ell}$, where $\tau(\cdot)$ will be discussed in more detail in Section III-C.

When the candidate sample selection is applied from $\mathbf{P}_{7}$ to $\mathbf{P}_{3}$, we only keep the $k$ samples that are closest to the center point of $g_{i}$ as positive samples based on L2 distance, which is denoted as top-$k$ in this paper. Here $k$ is a hyperparameter that ensures a balanced number of positive samples for each target as possible, which is suggested in [10] that unbalanced positive samples will decrease the performance, and L2 distance indicates the Euclidean Distance. In details, let $\mathcal{S}_{\ell}$ denote selection set on $\mathbf{P}_{\ell}$ and $\tilde{n}_{\mathcal{S}}^{\ell}$ denote the number of selected samples on $\mathbf{P}_{\ell}$, which is formulated as:

where $\tilde{n}_{\mathcal{R}}^{\ell}$ denotes the number of candidate samples on $\mathbf{P}_{\ell}$. Finally, we subtract $\tilde{n}_{\mathcal{S}}^{\ell}$ from $k$ and add the selected candidate samples to the positive samples set $\mathcal{P}$, after which we repeat this operation if $k$ is greater than 0. Algorithm 1 specifies the proposed method for positive sample selection.

As a result, more high-level samples and fewer low-level samples are assigned to the large targets, and the opposite is conducted for the small targets, according to the scales of targets, while the number of ambiguous samples can be reduced adaptively. In addition, if an anchor point is still assigned to multiple ground truths, it is forced to be assigned to the one with the longest side. Besides, as shown in Fig. 5, if a ground truth escapes from any samples, we assign it with the closest unassigned anchor point, which ensures the information from each target can be used reasonably and partially alleviates the insufficient sampling problem.

In this way, the sequentially adaptive sampling operation of ADS can obtain a better distribution of positive samples when compared with the fixed scale assignment (FSA) strategy and the top-$k$ samples for each target (TKT) strategy (i.e., ADS without the sequential selection). As shown in Fig. 6, FSA strategy constrains the level distribution of positive samples. Whereas TKT strategy selects samples clustered at the bottom levels of feature maps, resulting in the imbalanced scale-level sample distribution. Therefore, above strategies will lead to the scale-level bias, especially when the target scale distribution is unbalanced, which is a common phenomenon in RSIs. In contrast, our proposed ADS can alleviate the problems mentioned above by sequentially sampling from high to low level of feature maps adaptively according to the scale variation of targets in RSIs.

In the Section III-B, the Algorithm 1 has a determined function $\tau(\cdot)$ that takes the ground truth $g_{i}$ and the anchor point $\alpha_{j}^{\ell}$ as input, and returns the true or false value depending on whether $\alpha_{j}^{\ell}$ is inside the certain region of $g_{i}$ or not. Here, the determined function achieves spatial assignment by constraining the sampling range. Concretely, existing methods [19, 30] often adopt the central area or the rectangle bounding box as the sampling range for spatial assignment, as shown in Fig. 7 (b) and (c), respectively. However, these methods rarely consider the characteristics of targets in RSIs, which either introduce more background noise or result in insufficient sampling.

To this end, the proposed DED strategy selects samples following an elliptical distribution as illustrated in Fig. 7 (a), which can adjust sampling range dynamically according to the aspect ratios of targets, i.e., the sampling range tends to be a contracted circular distribution when the shape of target is close to a square, and an inner tangent ellipse when the shape is elongated. Such that it can achieve more flexible sample distribution to fit the shapes and orientations of targets.

Suppose that given a ground truth $g_{i}$ and an anchor point $\alpha_{j}^{\ell}$, which is mapped back to the input image, the function $\tau(\cdot)$ can be formulated as:

where $a$ and $b$ are calculated by the angle of $g_{i}$ and the offsets of coordinates between $g_{i}$ and $\alpha_{j}^{\ell}$, which are formulated respectively as:

the ratio factor $\xi$ is an adaptive threshold determined by the target shape that controls the range of elliptical distribution, and to select high-quality samples with less background noise. Here, $\xi$ is calculated as follows:

In this way, by considering the shapes and orientations of the targets, our proposed DED strategy can avoid introducing too much background noise, which is more suitable for object detection in RSIs.

To further mitigate the effect of low-quality predictions generated from the anchor points far away from the center of ground truth, instead of employing extra prediction branch such as centerness branch[11] or IoU prediction branch[19], SDW module is proposed to enhance the high-quality sample selection by assigning different weights to the samples at different pixel-wise locations.

For each positive sample $\alpha_{j}^{\ell}\in\mathcal{P}$, SDW module calculates weight $\mathcal{W}_{j}$ according to the scale of $g_{i}$ that it belongs to, and the L2 distance between $\alpha_{j}^{\ell}$ and center point of $g_{i}$, which is represented as $d_{\alpha_{j}^{\ell},g_{i}}$. For classification, $\mathcal{W}_{j}$ is used to weight the probability for the corresponding category. For box regression, we first establish the regression loss and then multiply by $\mathcal{W}_{j}$. Here, $\mathcal{W}_{j}$ is formulated as:

where $m=max(\frac{h_{i}}{2},\frac{w_{i}}{2})$, and $d_{\mathcal{P},g_{i}}$ is the set of all the distance of positive samples for $g_{i}$. The weight is normalized as in Eq. (7) to make sure that the highest weight is given to the closest positive sample, and achieves adaptive distance weighting, i.e., assigning larger weights to the samples close to the center of the target, and smaller weights to the samples near the boundaries.

Finally, $\mathcal{W}_{j}$ is applied to weight the classification probability and box regression loss. The total loss $\mathcal{L}_{total}$ is constructed with two types of loss, i.e., the classification loss $\mathcal{L}_{cls}$ and box regression loss $\mathcal{L}_{reg}$, denoted as

where $\lambda$ is the penalty parameter to balance these two types of loss, $N_{pos}$ denotes the number of positive samples, which is used for normalization. $\mathds{1}_{\{condition\}}$ is the indicator function, which equals to 1 if the condition is satisfied, otherwise 0. The classification loss is formulated as:

where $c^{*}_{j}\in\{0,1\}$ is the ground truth for classification and $\hat{c_{j}}\in[0,1]$ is the classification score from the network, $\delta$ and $\gamma$ are hyperparameters of the focal loss[10]. The box regression loss is formulated as:

where $Smooth_{L1}(\cdot)$ is smooth L1 loss function as defined in [47].

With SDW module to enhance the high-quality sample selection, our proposed EARL strategy can further improve the detection performance.

The sample number $k$ in ADS strategy is empirically selected in our study. Due to its importance, we conducted experiments on two complex datasets, i.e., DOTA and DIOR-R, to quantitatively analyze the impact of the choice of $k$ on detection performance. In addition, we found that the number of samples affects the training duration to some extent, so for the sake of efficiency, we test $k$ from 12 to 18, and the results are given in Fig. 8.

For the performance on DOTA observed from Fig. 8, when $k$=12, although the quality of each selected sample is guaranteed, the number of positive samples is inadequate for the model to learn feature distribution, so the mAP has only reached 71.52%. The performance of the model gradually improves as $k$ increases, and the detector achieves the best mAP of 72.87% when $k$=15. However, as $k$ continues to increase, more background noise are included since samples located near the boundaries of targets are selected, so the mAP score drops from 72.87% to 71.93% when $k$=18. Whereas for the selection of $k$ on DIOR-R, it shows the similar performance trend as that on DOTA, where the best performance is obtained when $k$ is set as 14. It can be seen that various $k$ selections yield relatively stable detection performance and can be simply changed the value of $k$ to suit different datasets.

In this section, a series of ablation experiments were conducted on three popular datasets, i.e., DOTA, DIOR-R and HRSC2016, and the detailed results are listed in Tables I, II and III, respectively. In addition, we also visualized the gradient-based feature heatmaps[56] to further understand the effectiveness of each component of EARL. The visualization results are illustrated in Fig. 11. Here, the components of proposed EARL are indicated in abbreviated form, i.e., ‘-A’ indicates ADS strategy, ‘-E’ refers to DED strategy, and ‘-W’ means SDW module. In addition, ‘w/o’ is applied to indicate without specified modules.

To further justify the effectiveness and superiority of the proposed scale assignment strategy, i.e., ADS, and the spatial assignment strategy, i.e., DED, a series of experiments were conducted on DOTA and DIOR-R datasets. For fair comparison, all compared label assignment strategies are evaluated on the same baseline detector (i.e., R-FCOS), and the results are listed in Tables IV, V and VI.

To evaluate the performance of the proposed EARL strategy, we conducted experiments compared with the state-of-the-art methods on three public RSIs datasets, i.e., DOTA, DIOR-R and HRSC2016, and the results are listed in Tables VII, VIII and IX, respectively. Note that, for the purpose of fair comparison, these results are from the corresponding papers of the comparison methods and datasets, unless specified. In addition, to fully validate the effectiveness and robustness of the proposed EARL strategy, a simple detector, i.e., R-FCOS, and an advanced detector, i.e., RTMDet-R, are selected as the baseline methods, where our EARL is deployed with them, denoted as R-FCOS-EARL and RTMDet-R-EARL, respectively.