ARS-DETR: Aspect Ratio-Sensitive Detection Transformer for Aerial Oriented Object Detection

As an emerging task, oriented object detection has made great progress in recent years. The simple solution [23] for oriented object detection task is to change the Anchor or Region of Interests (RoIs) from the horizontal type to the rotated type. RoI-Transformer [4] constructs the geometry transformation to rotate the proposals to locate the objects more accurately. To address the feature misalignment in refined single-stage detectors, R³Det [24] and S²A-Net [25] adopt the feature alignment module to get a more accurate location. However, these mainstream regression-based methods often suffer the boundary problem [8] due to the predictions beyond the defined range and need additional complicated treatment. SCRDet [5] designs a novel IoU-Smooth L1 Loss to alleviate the sudden increase in loss caused by angle periodicity and edge exchangeability, which reduces the difficulty of model training.

CSL [8] transforms the prediction of angle from regression to classification, thereby eliminating the boundary problem. It is achieved through the design of a Circle Smooth Label. Gliding vertex [6] glides the vertex of the horizontal bounding box to accurately represent a multi-oriented object. GWD [11], KLD [26] and KFIoU [27] convert the rotated bounding box into a Gaussian distribution to avoid the boundary discontinuity and square-like issue in oriented object detection. PSC [28] provides a unified framework for various periodic fuzzy issues in oriented object detection by mapping rotational periodicity of different cycles into phase of different frequencies.

The classification-based angle prediction method is a novel and effective approach to circumvent the boundary problem while predicting angle accurately, and it has also made a lot of progress. CSL[8] discretizes the angle variable into 180 categories and smooths the angle label via a Gaussian window function. The CSL method directly promotes the development of classification-based oriented object detection algorithms. DCL [9] uses dense coded label to reduce the amount of computation and parameters of CSL. [10] adopts a dynamic weighting mechanism based on CSL to perform precise angle estimation for rotated objects. To overcome the challenges of ambiguity and high costs in angle representation, [29] proposes a multi-grained angle representation method, consisting of coarse-grained angle classification and fine-grained angle regression. TIOE [12] proposes a progressive orientation estimation strategy to approximate the orientation of objects with n-ary codes. AR-BCL [30] uses an aspect ratio-based bidirectional coded label to solve the square-like detection issue [9]. In contrast, the new angle encoding technique proposed in this paper is free from hyperparameters, boundary problem, and square-like issue. Furthermore, it explores the potential of angle classification in high-precision detection, which is overlooked by most of the above methods.

DETR [13] proposed a Transformer-based end-to-end object detector without using hand-designed components like prior anchor design and NMS. In recent years, DETR has progressed a lot and also exhibits its strong ability in object detection compared with classic detection methods [14, 16, 17, 18, 19, 20]. Deformable DETR [14] proposes a deformable attention module to sample the value of adaptive positions around the reference point and utilizes the multi-level features to mitigate the slow convergence and high complexity issues of DETR. DAB-DETR [20] provides explicit positional priors for each query to let the cross-attention module focus on a local region corresponding to a target object by using anchor box size. DN-DETR [17] and DINO [18] design a denoising auxiliary task that bypasses the bipartite graph matching. This not only accelerates training convergence but also leads to a better training result. In addition, there have been some DETR-based oriented object detectors [21, 22]. O²DETR [21] is the first attempt to apply DETR to the oriented object detection task and AO2-DETR [22] introduces oriented proposal generation and refinement module into the transformer architecture to refine the features. Nevertheless, both of them predict angles using a simple regression way and do not address boundary discontinuity or embed angle information into DETR.

Objects with different aspect ratios have different sensitivities to angles. In order to better observe the relationship between the angle and aspect ratio, we assume that there are two bounding boxes with the same center, width and height, and give the SkewIoU of these two boxes with different aspect ratios under different angle deviations, as shown in Fig. 2. The SkewIoU can be calculated by Eq. 1:

where $k\geq 1$ is the aspect ratio, and $\Delta\theta\in[0^{\circ},90^{\circ}]$ represents angle deviation, indicating the absolute value of the angle difference between two boxes. There is a critical angle boundary threshold ($\theta^{*}=2\arctan\frac{1}{k}$) as shown in Fig. 3. The symmetrical nature depicted in Fig. 2 demonstrates that variations in $\Delta\theta$ exhibit bidirectional characteristics.

Fig. LABEL:sub@fig:skewiou1 shows the curves between SkewIoU and angle deviation under different aspect ratios. It can be seen that the SkewIoU variation trends of bounding boxes with different aspect ratios are obviously divided into two types according to metric AP₅₀ (if the SkewIoU between the prediction box and ground truth is greater than 0.5, then it will be judged as true positive), and the dividing boundary is $k=1.5$, as shown in Fig. LABEL:sub@fig:skewiou2 ($1\leq k\leq 1.5$) and Fig. LABEL:sub@fig:skewiou3 ($k>1.5$), respectively. Specifically, Fig. LABEL:sub@fig:skewiou2 shows that when the aspect ratio is smaller than 1.5, SkewIoU is always greater than 0.5 regardless of the angle deviation (see pink dashed line). In contrast, when the aspect ratio is greater than 1.5, as shown in Fig. LABEL:sub@fig:skewiou3, SkewIoU will decay rapidly with the increase of angle deviation, but the valid angle deviation still retains a wide range. In summary, objects with a small aspect ratio are less sensitive to angle deviation, whereas objects with a large aspect ratio are more sensitive but still exhibit a significant tolerance for angle deviation under AP₅₀.

Considering that angle is a very important parameter in oriented object detection, the accuracy of its estimation will greatly affect the subsequent related tasks, such as object fine-grained recognition [31], object heading estimation [32, 33], etc., AP₅₀ seems not accurate enough for reflecting the performance of high-precision oriented object detection. Therefore, we advocate to use more stringent metric, e.g. AP₇₅¹¹1Note: The difficulty of achieving high AP₇₅ in oriented object detection is more difficult than that in horizontal object detection, because the rotated bounding box is more accurate with less redundant areas, thus more sensitive to errors., which is usually used in generic detection, to measure this challenging task. Under the AP₇₅ metric, not only are the prediction boxes required to be closer to the ground truth boxes, but the angle prediction requirement is also more stringent. As shown the gray dashed line in Fig. 2, when AP₇₅ is adopted, regardless of the aspect ratio, the angle deviation should be controlled within a specific range; otherwise, they will not be judged as positive. The larger the aspect ratio, the narrower the range.

Tab. I compares the accuracy of some oriented object detectors using AP₅₀ and AP₇₅, respectively. It can be seen that all detectors achieve a high performance in terms of AP₅₀ and the gap among them is small. However, the situation becomes different when AP₇₅ is used, some detectors may not good as other detectors whose performance on AP₅₀ are lower than them, e.g. S²A-Net vs. Rotated ATSS. Therefore, AP₇₅ can further represent the performance of high-precision oriented object detection.

As for the AP_50:95, which is also often adopted in generic detection, it is no doubt a stricter and more comprehensive metric, but we believe that AP₇₅ could reflect high-precision oriented object detection more directly. AP_50:95 contains a large number of metrics. Under the metrics AP₅₀ or AP₅₅, when the angle prediction is not accurate, as shown in Fig. 1, a relatively high value could still be reached. Conversely, under the metrics AP₉₀ or AP₉₅, the performance of oriented detectors degenerates a lot, which is of no significance for comparison in current research. Therefore, AP₇₅, as an intermediate metric, is more balanced and suitable for high-precision oriented object detection. Meanwhile, there is an average operation in AP_50:95, which makes metrics like AP₅₀ and AP₅₅ have a great influence on the overall value.

In summary, it is necessary and meaningful to pay more attention to high-precision oriented object detection.

Fig. 4 shows the framework of the proposed ARS-DETR. Given an image, a backbone is firstly used to extract feature maps. The backbone will generate hierarchical feature maps and the last three stages of outputs are used. Then, the 1$\times$1 convolution is adopted to map their channels to the uniform dimension. Additionally, the lowest resolution feature map is obtained via a 3$\times$3 convolution on the final feature map. Then the multi-scale feature maps, embed with the 2D positional encoding, are fed into Encoder. Without the use of top-down structure in FPN[36], multi-scale Deformable Attention can exchange the information among multi-scale feature maps and further refine these feature maps. The output of Encoder will then generate a large number of proposals and be used in Decoder. Next, the Top-K scoring proposals are picked as object queries and transformed into output embeddings by MultiHead Attention and multi-scale Rotated Deformable Attention in Decoder. Finally, the prediction head further decode the output embeddings from decoder into class labels, angle labels and horizontal box coordinates. Additionally, during the training process, we utilize the ground truth with noise as queries to participate in the training to stabilize the training. At the same time, we adopt Aspect Ratio sensitive Matching (ARM) to adjust the influence of angle in matching process and adopt Aspect Ratio sensitive Loss (ARL) to adjust the training strategy for different aspect ratio objects when calculating angle loss.

Fig. LABEL:sub@fig:architecture1 shows a simple DETR-based oriented detector [21, 22]. This detector merely adds an additional angle parameter in the prediction head to accomplish rotated bounding box estimation. Nevertheless, it fails to embed the angle information into the detector to exploit the maximum potential of the detector, resulting in feature misalignment. To address this, we present a Rotated Deformable Attention module (RDA).

Given an input feature map $x\in\mathbb{R}^{C\times H\times W}$, let $q\in\Omega_{q}$ index a query element with representation feature $z_{q}\in\mathbb{R}^{C}$ and reference box $b_{q}=\left[p_{q},w_{q},h_{q},\theta_{q}\right]=\left[(x_{q},y_{q}),w_{q},h_{q},\theta_{q}\right]$, where $p_{q}=(x_{q},y_{q})\in\left[0,1\right]^{2}$ is the centric point of the reference box and $w_{q}\in\left[0,1\right]$, $h_{q}\in\left[0,1\right]$, $\theta_{q}\in\left[-\frac{\pi}{4},\frac{\pi}{4}\right]$ are the width, height and angle of the reference box respectively. The rotated deformable attention feature is calculated as follows:

where $m$ indexes the attention head, and $k$ indexes the sampled points. $M$ is the total head number and $K$ is the total sampled points number. We use $M=8$ and $K=4$ following [14]. $W_{m}^{{}^{\prime}}\in\mathbb{R}^{\frac{C}{M}\times C}$ and $W_{m}\in\mathbb{R}^{C\times\frac{C}{M}}$ are learnable weights. $\Delta p_{mqk}$ and $A_{mqk}$ denote the sampling offset and attention weight of the $k^{th}$ sampling point in the $m^{th}$ attention head, respectively. The scalar attention weight $A_{mqk}$ lies in the range $\left[0,1\right]$, normalized by $\sum_{k=1}^{K}A_{mqk}=1$. For each $\Delta p_{mqk}$, it is calculated by:

where $f_{mk}\in\mathbb{R}^{C\times 2}$ is the projection matrices. $r_{mk}$ is the bias of the $k^{th}$ sampling point in the $m^{th}$ attention head and is calculated by:

$R(\theta_{q})=(cos\theta_{q},-sin\theta_{q};sin\theta_{q},cos\theta_{q})^{T}$ is the rotation matrix.

In this way, we can obtain the dynamic sampling points $p_{q}+\Delta p_{mqk}$ and constrain them as much as possible within $b_{q}$ to extract aligned features.

As depicted in Fig. LABEL:sub@fig:rda1, the sampling points in Deformable Attention Module will be adjusted according to the corresponding reference box, so that the sampling points will be restricted within the reference box and fall within the object as far as possible. However, as shown in Fig. LABEL:sub@fig:rda2, when the object is of the oriented type, the sampling points cannot accurately align with the object if the horizontal reference box [22] is still used. Therefore, we design the Rotated Deformable Attention Module to align the sampling points with features by rotating the sampling points according to the embedded angle information, as shown in Fig. LABEL:sub@fig:rda3 and Fig. LABEL:sub@fig:rda4. Moreover, instead of refining the angle layer by layer, we predict a new angle after each layer independently, as shown in Fig. LABEL:sub@fig:architecture2.

As shown in Fig. 8, we compare two other sampling methods with our RDA. Fig. LABEL:sub@fig:deformable_offsets shows the sampling method in [37]. It learns deformable offsets to augment the spatial sampling locations, but it may sample from wrong locations with weak supervision, especially for densely packed objects. Fig. LABEL:sub@fig:fixed_offsets shows the sampling methods in [4, 25]. It rotates the sampling points with the angle of the bounding box to align features, but the sampling positions are fixed. Our RDA also rotates the sampling points but it also learns deformable offsets and will constrain them according to the width and height of the bounding box. Compared with these methods, RDA provides some flexibility while aligning features.

It is verified in [17] that, the instability of bipartite graph matching in DETR could result in slow convergence and hence hinder the performance. The proposed DETR-based detection learns to refine the coarse object features and boxes iteratively, which can be simulated by the process of reconstructing noisy ground-truth labels and boxes. Denoising Training, as an auxiliary task of denoising labels and boxes, has fixed target assigning results, so it can mitigate the effect of matching instability and accelerate the convergence. Besides, to simulate predicting both positive and negative samples, both positive and negative noisy targets are generated for each ground-truth target, which provides a more reasonable optimization goal. Additionally, to transfer the Denoising Training to the oriented object detection, we also design the task of denoising angles for training procedure.

Given a ground truth $gt=(c_{gt},b_{gt},\theta_{gt})$, where $c_{gt}$ is the class, $b_{gt}=(x_{c},y_{c},w,h)$ is the bounding box, and $\theta_{gt}$ is the angle, the noisy ground truth $gt_{n}=(c_{n},b_{n},\theta_{n})$ is obtained as follows.

The noisy labels $c_{n}$ are generated by randomly selecting part of the ground-truth labels and overlaying the selected labels with arbitrary object labels at a ratio of $\alpha$. The target labels of positive samples are assigned with the ground-truth labels, while those of negative samples are assigned with the background category.

The noisy boxes $b_{n}$ are generated by moving the four boundaries of ground-truth boxes randomly. Specifically, the ground truth box $b_{gt}=(x_{c},y_{c},w,h)$ is convert into format $\hat{b}_{gt}=(x_{l},y_{u},x_{r},y_{b})$, where $x_{l}$ and $x_{r}$ represent horizontal ordinates of left and right boundaries, respectively, and $y_{u}$ and $y_{b}$ represent vertical ordinates of upper and bottom boundaries, respectively. The negative noisy boxes should have larger noise scale than positive ones, because the farther proposals should predict negative samples [18]. Hence, random noise offsets $\hat{\epsilon}=(\Delta x_{l},\Delta y_{u},\Delta x_{r},\Delta y_{b})$ where $\Delta x_{l},\Delta x_{r}\sim U(-\frac{\beta}{2}w,\frac{\beta}{2}w)$ and $\Delta y_{u},\Delta y_{b}\sim U(-\frac{\beta}{2}h,\frac{\beta}{2}h)$, are generated for positive noisy boxes. While for negative ones, $\Delta x_{l},\Delta x_{r}\sim\left(U(-\beta w,-\frac{\beta}{2}w)+U(\frac{\beta}{2}w,\beta w)\right)$ and $\Delta y_{u},\Delta y_{b}\sim\left(U(-\beta h,-\frac{\beta}{2}h)+U(\frac{\beta}{2}h,\beta h)\right)$. The noisy boxes are calculated with $\hat{b_{n}}=\hat{b_{gt}}+\hat{\epsilon}$ and then converted into format $b_{n}=(x_{nc},y_{nc},w_{n},h_{n})$ as initial box proposals. The decoder learns to denoise $b_{n}$ and reconstruct $b_{gt}$.

The noisy angles $\theta_{n}$ are generated by shifting the $\theta_{gt}$ with $\theta_{n}=f(\theta_{gt}+\Delta\theta)$, where $\Delta\theta\sim U(-\gamma\pi,\gamma\pi)$ are generated for positive noisy angles and $\gamma$ is the angle noise scale. While for negative ones, $\Delta\theta\sim U(-2\gamma\pi,2\gamma\pi)$. $f()$ is a periodic function, thereby ensuring that the $\theta_{gt}+\Delta\theta$ remain within the defined range.

After decoding the output embeddings from decoder, the predicted results will be matched with targets to count the loss during the training. Let $y$ denote ground truth set of oriented objects and $\hat{y}$ denote the $N$ predictions. Then we calculate the cost between these two sets and search for a permutation of $N$ elements $\sigma\in{{O}_{n}}$ with the lowest cost:

where ${{L}_{match}}({{y}_{i}},{{\hat{y}}_{\sigma(i)}})$ is a pair-wise matching cost between ground truth ${{y}_{i}}$ and a prediction with index $\sigma(i)$, which takes into account the class prediction, angle prediction and the similarity of predicted and ground truth horizontal boxes, and we define it as follows:

where the $c$ is the class label, $b$ is the horizontal box, and $\theta$ is the angle, respectively. Additionally, the ${{L}_{cls}}$ is the focal loss, the ${{L}_{box}}$ is the L1 loss, the ${{L}_{iou}}$ is the generalized intersection over union loss (GIoU), and ${{L}_{\theta}}$ is the cross entropy loss.

Whereas, considering the objects with larger aspect ration are more sensitive to the angle, we introduce a dynamic coefficient to adjust the angle loss, named Aspect Ratio sensitive Matching (ARM), which can be formulated as follows:

where $k_{i}$ is the aspect ratio of the targets and $k_{i}\geq 1$.

As shown in Fig. 9, the red ground truth will be matched to the blue prediction whose cost is the lowest according to Eq. 8. When the ARM is introduced, the ground truth with a large aspect ratio will consider more about the angle deviation in the cost, so it is prone to be matched to the prediction whose angle is more similar with its.

The definition of training loss function is consistent with the Eq. 8. Meanwhile, we also introduce the Aspect Ratio sensitive Loss (ARL), which is the same as Eq. 9, to dynamically adjust the angle loss.

DOTA-v1.0 [2] is one of the largest datasets for oriented object detection, containing 2,806 large aerial images from different sensors and platforms ranging from around 800 $\times$ 800 to 4,000 $\times$ 4,000 and 188,282 instances. It has 15 common categories: Plane (PL), Baseball diamond (BD), Bridge (BR), Ground track field (GTF), Small vehicle (SV), Large vehicle (LV), Ship (SH), Tennis court (TC), Basketball court (BC), Storage tank (ST), Soccer-ball field (SBF), Roundabout (RA), Harbor (HA), Swimming pool (SP), and Helicopter (HC). We use both training and validation sets for training, and the test set for testing. We divide the images into 1024 $\times$ 1024 sub-images with an overlap of 200 pixels. During the training, only the random horizontal, vertical, diagonal flipping is adopted to avoid over-fitting and no other tricks are utilized. The performance of the test set is evaluated on the official DOTA evaluation server.

DIOR-R [3] is an aerial image dataset annotated by oriented bounding boxes from the DIOR [38] dataset. There are 23,463 images and 192,518 instances in this dataset, containing 20 common categories: Airplane (APL), Airport (APO), Baseball Field (BF), Basketball Court (BC), Bridge (BR), Chimney (CH), Expressway Service Area (ESA), Expressway Toll Station (ETS), Dam (DAM), Golf Field (GF), Ground Track Field (GTF), Harbor (HA), Overpass (OP), Ship (SH), Stadium (STA), Storage Tank (STO), Tennis Court (TC), Train Station (TS), Vehicle (VE) and Windmill (WM). DIOR-R has a high variation of object size, both in spatial resolutions, and in the aspect of inter-class and intra-class size variability across objects. Different imaging conditions, weather, seasons, image quality are the major challenges of DIOR-R. We use both training and validation sets for training and the test set for testing.

OHD-SJTU [33] is a public dataset for oriented object detection and object heading detection. It contains two different scale datasets, called OHD-SJTU-S and OHD-SJTU-L. OHD-SJTU-S collects 43 large scene images ranging from 10,000 $\times$ 10,000 to 16,000 $\times$ 16,000 and 4,125 instances in this dataset. In contrast, OHD-SJTU-L adds more categories and instances, containing six object categories and 113,435 instances. In line with previous work [24, 33] processing, we divide the images into 600 $\times$ 600 sub-images with an overlap of 150 pixels and scale them to 800 $\times$ 800.

All models in this paper are implemented by PyTorch [39] based framework MMRotate [40], and trained with AdamW [41] optimizer. The initial learning rate is 10^-4 with 2 images per mini-batch with ‘3x’ training schedule on DOTA-v1.0, DIOR-R and OHD-SJTU-L and ‘9x’ training schedule on OHD-SJTU-S respectively. In addition, we adopt learning rate warm-up for 500 iterations, and the learning rate is divided by 10 at decay step.

Results on DOTA-v1.0. We report the results of 16 oriented object detectors in Tab. X. Since different methods use different image resolutions with different data pre-processing, data augmentation, backbone, training strategies, various tricks and etc. in the original papers, we implement all detectors on MMRotate [40], using the same setting to make the comparison as fair as possible. All the results are obtained by single-scale training and testing, and adopted the ‘1x’ (12epochs) or ‘3x’ (36 epochs) training schedule. With R-50 and Swin-T as the backbone, our method obtains 74.16% and 75.47% on AP₅₀, and 49.41% and 51.77% on AP₇₅, respectively, as shown in Fig. 12. When using the ResNet50 as backbone, the performance of ARS-DETR under AP₅₀ is not as good as that of many advanced oriented detectors, but it has obvious advantages in high-precision detection and surpasses other advanced detectors on AP₇₅. Specially, ARS-DETR outperforms RoI Trans by 0.55% (49.41% VS 48.86%), Oriented Reppoints by 2.85% (49.41% VS 46.56%), CFA by 2.86% (49.41% VS 46.55%), GWD by 4.2% (49.41% VS 45.21%). In addition, there are also some detectors perform well on AP₅₀ but degenerate a lot on AP₇₅ (e.g. S²A-Net, 75.29% on AP₅₀ and 40.08% on AP₇₅) and some detectors are relatively not good at AP₅₀ but has a favorable performance on AP₇₅ (e.g. PSC, 72.87% on AP₅₀ and 46.18% on AP₇₅), which further proves that it is not suitable to only use AP₅₀ to judge the performance of the detector. What’s more, ARS-DETR also exceeds DETR-based detectors like Rotated Deformable DETR and AO2-DETR.

Results on DIOR-R. Results on DIOR-R dataset are shown in Tab. XI. All methods adopt ‘3x’ training schedule and use R-50 as backbone. ARS-DETR achieves 66.12% and 45.81% on AP₅₀ and AP₇₅, respectively. The visualization is shown in Fig. 13.

Results on OHD-SJTU. We also compare the performance of some oriented object detection methods on OHD-SJTU, mainly include R2CNN, RRPN, RetinaNet, R³Det, OHDet. Without any bells and whistles, our ARS-DETR achieves 46.08% and 80.67% on AP₇₅ in OHD-SJTU-L and OHD-SJTU-S respectively, surpassing other advanced oriented object detectors. The detailed results are shown in Tab. XII and Tab. XIII.

Besides, from the above comparisons, it can be observed that AP₅₀ is not accurate enough to represent the performance of oriented object detectors, especially the performance of high-precision oriented detection. In short, this paper mainly advocates the use of more stringent indicators (e.g. AP₇₅) to further study high-precision oriented object detector.