In H2RBox-v2, SS branch is designed to learn the orientation of objects from symmetry via consistencies between several views of training images, whilst WS branch learns other properties from the HBox annotation. The core idea of symmetry-aware SS learning is described below.

Definition of reflection symmetry: An object has reflectional symmetry if there is a line going through it which divides it into two pieces that are mirror images of each other [58].

Assume there is a neural network $f_{\text{nn}}\left(\cdot\right)$ that maps a symmetric image $I$ to a real number $\theta$:

$$\theta=f_{\text{nn}}\left(I\right)$$

(1)

To exploit reflection symmetry, we endow the function with two new properties: flip consistency and rotate consistency.

Property I: Flip consistency. With an input image vertically flipped, $f_{\text{nn}}\left(\cdot\right)$ gives an opposite output:

$$f_{\text{nn}}\left(I\right)+f_{\text{nn}}\left(\text{flp}\left(I\right)\right)=0$$

(2)

where $\text{flp}\left(I\right)$ is an operator of vertically flipping the image $I$.

Property II: Rotate consistency. With an input rotated by $\mathcal{R}$, the output of $f_{\text{nn}}\left(\cdot\right)$ also rotates by $\mathcal{R}$:

$$f_{\text{nn}}\left(\text{rot}\left(I,\mathcal{R}\right)\right)-f_{\text{nn}}\left(I\right)=\mathcal{R}$$

(3)

where $\text{rot}\left(I,\mathcal{R}\right)$ is an operator that clockwise rotates the image $I$ by $\mathcal{R}$.

Now we consider that given an image $I_{0}$ symmetric about $\varphi=\theta_{\text{sym}}$, assuming the corresponding output is $\theta_{\text{pred}}=f_{\text{nn}}\left(I_{0}\right)$, the image can be transformed in two ways as shown in Fig. 3:

$\bullet$ Way 1: Flipping $I_{0}$ along line $\varphi=\theta_{\text{sym}}$. According to the above definition of reflection symmetry, the output remains the same, i.e. $f_{\text{nn}}\left(I^{\prime}\right)=f_{\text{nn}}\left(I_{0}\right)=\theta_{\text{pred}}$.

$\bullet$ Way 2: Flipping $I_{0}$ vertically to obtain $I_{1}$ first, and then rotating $I_{1}$ by $2\theta_{\text{sym}}$ to obtain $I_{2}$. According to flip and rotate consistencies, the output is supposed to be $f_{\text{nn}}\left(I_{2}\right)=-\theta_{\text{pred}}+2\theta_{\text{sym}}$.

On the ground that ways 1 and 2 are equivalent, the transformed images $I^{\prime}$ and $I_{2}$ are identical. And thus $f_{\text{nn}}\left(I^{\prime}\right)=f_{\text{nn}}\left(I_{2}\right)$, finally leading to $\theta_{\text{pred}}=\theta_{\text{sym}}$.

From the above analysis, it arrives at a conclusion that if image $I_{0}$ is symmetric about line $\varphi=\theta_{\text{sym}}$ and function $f_{\text{nn}}\left(\cdot\right)$ subjects to both flip and rotate consistencies, then $f_{\text{nn}}\left(I_{0}\right)$ must be equal to $\theta_{\text{sym}}$, so the orientation is obtained. In the following, we further devise two improvements.

Handling the angle periodicity: For higher conciseness, the periodicity is not included in the above formula. To allow images to be rotated into another cycle, the consistencies are modified as:

	$\displaystyle f_{\text{nn}}\left(I\right)+f_{\text{nn}}\left(\text{flp}\left(I\right)\right)=$	$\displaystyle k\pi$		(4)
	$\displaystyle f_{\text{nn}}\left(\text{rot}\left(I,\mathcal{R}\right)\right)-f_{\text{nn}}\left(I\right)=$	$\displaystyle\mathcal{R}+k\pi$

where $k$ is an integer to keep left and right in the same cycle. This problem is coped with using the snap loss in Sec. 3.4. Meanwhile, the conclusion should be amended as: $f_{\text{nn}}\left(I_{0}\right)=\theta_{\text{sym}}+k\pi/2$, meaning that the network outputs either the axis of symmetry or a perpendicular one.

Extending to the general case of multiple objects: Strictly speaking, our above discussion is limited to the setting when image $I_{0}$ contains only one symmetric object. In detail implementation, we use an assigner to match the center of objects in different views, and the consistency losses are calculated between these matched center points. We empirically show on several representative datasets (see Sec. 4.2) that our above design is applicable for multiple object detection with objects not perfectly symmetric, where an approximate axis of each object can be found via learning.

The above analysis suggests that the network can learn the angle of objects from symmetry through the flip and rotate consistencies. A self-supervised branch is accordingly designed.

During the training process, we perform vertical flip and random rotation to generate two transformed views, $I_{\text{flp}}$ and $I_{\text{rot}}$, of the input image $I$, as shown in Fig. 2 (a). The blank border area induced by rotation is filled with reflection padding. Afterward, the three views are fed into three parameter-shared branches of the network, where ResNet50 [59] and FPN [60] are used as the backbone and the neck, respectively. The random rotation is in range $\pi/4\sim 3\pi/4$ (according to Table 9).

Similar to H2RBox, a label assigner is required in the SS branch to match the objects in different views. In H2RBox-v2, we calculate the average angle features on all sample points for each object and eliminate those objects without correspondence (some objects may be lost during rotation).

Following the assigner, an angle coder PSC [36] is further adopted to cope with the boundary problem. Table 6 empirically shows that the impact of boundary problem in our self-supervised setting could be much greater than that in the supervised case, especially in terms of stability. The decoded angles of the original, flipped, and rotated views are denoted as $\theta$, $\theta_{\text{flp}}$, and $\theta_{\text{rot}}$.

Finally, with formula in Sec. 3.4, the loss $L_{\text{flp}}$ can be calculated between $\theta$ and $\theta_{\text{flp}}$, whereas $L_{\text{rot}}$ between $\theta$ and $\theta_{\text{rot}}$. By minimizing $L_{\text{flp}}$ and $L_{\text{rot}}$, the network learns to conform with flip and rotate consistencies and gains the ability of angle prediction through self-supervision.

The SS branch above provides the angle of objects. To predict other properties of the bounding box (position, size, category, etc.), a weakly-supervised branch using HBox supervision is further introduced. The WS branch in H2RBox-v2 is inherited from H2RBox, but has some differences:

1) In H2RBox, the angle is primarily learned by the WS branch, with an SS branch eliminating the undesired results. Comparably, H2RBox-v2 has a powerful SS branch driven by symmetry that can learn the angle independently, and thus the angle subnet is deleted from the WS branch.

2) H2RBox converts $B_{\text{pred}}$ to an HBox to calculate the IoU loss [61] with the HBox $B_{\text{gt}}$, which cannot work with random rotation where $B_{\text{gt}}$ becomes an RBox. To solve this problem, CircumIoU loss (detailed in Sec. 3.4) is proposed in H2RBox-v2 to directly calculate $L_{\text{box}}$ between $B_{\text{pred}}$ and $B_{\text{gt}}$, so that $B_{\text{gt}}$ is allowed to be an RBox circumscribed to $B_{\text{pred}}$, as is shown in Fig. 2 (c).

Loss for SS branch: As described in Eq. (4), the consistencies encounter the periodicity problem in rotation. To cope with this problem, the snap loss $\ell_{s}$ ¹¹1There are a series of targets with interval $\pi$, just like a series of evenly spaced grids. The snap loss moves prediction toward the closest target, thus deriving its name from “snap to grid” function. is proposed in this paper:

$$\ell_{s}\left(\theta_{\text{pred}},\theta_{\text{target}}\right)=\min_{k\in Z}\left(smooth_{L1}\left(\theta_{\text{pred}},k\pi+\theta_{\text{target}}\right)\right)$$

(5)

where an illustration is displayed in Fig. 4 (a). The snap loss limits the difference between predicted and target angles to $\pi/2$ so that the loss calculation upon the two decoded angles will not induce boundary discontinuity. The snap loss is used in both $L_{\text{flp}}$ and $L_{\text{rot}}$ as:

$$\begin{matrix}L_{\text{flp}}=\ell_{\text{s}}\left(\theta_{\text{flp}}+\theta,0\right)\\ L_{\text{rot}}=\ell_{\text{s}}\left(\theta_{\text{rot}}-\theta,\mathcal{R}\right)\end{matrix}$$

(6)

where $L_{\text{flp}}$ is the loss for flip consistency, and $L_{\text{rot}}$ for rotate consistency. $\theta$, $\theta_{\text{flp}}$, and $\theta_{\text{rot}}$ are the outputs of the three views described in Sec. 3.2. $\mathcal{R}$ is the angle in generating rotated view.

With the above definitions, the loss of the SS branch can be expressed as:

$$L_{\text{ss}}=\lambda L_{\text{flp}}+L_{\text{rot}}$$

(7)

where $\lambda$ is the weight set to 0.05 according to the ablation study in Table 7.

Loss for the WS branch: The losses in the WS branch are mainly defined by the backbone FCOS detector, including $L_{\text{cls}}$ for classification and $L_{\text{cn}}$ for center-ness. Different from RBox-supervised methods, $B_{\text{gt}}$ in our task is an RBox circumscribed to $B_{\text{pred}}$, so we re-define the loss for box regression $L_{\text{box}}$ with CircumIoU loss as:

$$\ell_{p}\left(B_{\text{pred}},B_{\text{gt}}\right)=-\ln\frac{B_{\text{proj}}\cap B_{\text{gt}}}{B_{\text{proj}}\cup B_{\text{gt}}}$$

(8)

where $B_{\text{proj}}$ is the dashed box in Fig. 4 (b), obtained by projecting the predicted box $B_{\text{pred}}$ to the direction of ground-truth box $B_{\text{gt}}$.

The CircumIoU loss enables H2RBox-v2 to use random rotation augmentation to further improve the performance (see Table 2), which is not supported by H2RBox.

Finally, the loss of the WS branch can be expressed as:

$$L_{\text{ws}}=L_{\text{cls}}+\mu_{\text{cn}}L_{\text{cn}}+\mu_{\text{box}}L_{\text{box}}$$

(9)

where the hyper-parameters are set to $\mu_{\text{cn}}=1$ and $\mu_{\text{box}}=1$ by default.

Overall loss: The overall loss of the proposed network is the sum of the WS loss and the SS loss:

$$L_{\text{total}}=L_{\text{ws}}+\mu_{\text{ss}}L_{\text{ss}}$$

(10)

where $\mu_{\text{ss}}$ is the weight of SS branch set to 1 by default.

Although using a multi-branch paradigm in training (as shown in Fig. 2), H2RBox-v2 does not require the multi-branch or the view generation during inference.

Due to the parameter sharing, the inference only involves the forward propagation of backbone, angle head (from SS), and other heads (i.e. regression, classification, and center-ness from WS). The only additional cost of H2RBox-v2 during inference is the PSC decoding (compared to H2RBox). Thus, FCOS/H2RBox/H2RBox-v2 have similar inference speed (see Table 2).

DOTA [4]: DOTA-v1.0 contains 2,806 aerial images—1,411 for training, 937 for validation, and 458 for testing, as annotated using 15 categories with 188,282 instances in total. DOTA-v1.5/2.0 are the extended version of v1.0. We follow the default preprocessing in MMRotate: The high-resolution images are split into 1,024 $\times$ 1,024 patches with an overlap of 200 pixels for training, and the detection results of all patches are merged to evaluate the performance.

HRSC [5]: It contains ship instances both on the sea and inshore, with arbitrary orientations. The training, validation, and testing set includes 436, 181, and 444 images, respectively. With preprocessing by MMRotate, images are scaled to 800 $\times$ 800 for training/testing.

FAIR1M [67]: It contains more than 1 million instances and more than 40,000 images for fine-grained object recognition in high-resolution remote sensing imagery. The dataset is annotated with five categories and 37 fine-grained subcategories. We split the images into 1,024 $\times$ 1,024 patches with an overlap of 200 pixels and a scale rate of 1.5 and merge the results for testing. The performance is evaluated on the FAIR1M-1.0 server.

Experimental settings: We adopt the FCOS [26] detector with ResNet50 [59] backbone and FPN [60] neck as the baseline method, based on which we develop our H2RBox-v2. We choose average precision (AP) as the primary metric to compare with existing literature. For a fair comparison, all the listed models are configured based on ResNet50 [59] backbone and trained on NVIDIA RTX3090/4090 GPUs. All models are trained with AdamW [68], with an initial learning rate of 5e-5 and a mini-batch size of 2. Besides, we adopt a learning rate warm-up for 500 iterations, and the learning rate is divided by ten at each decay step. “1x”, “3x”, and “6x” schedules indicate 12, 36, and 72 epochs for training. “MS” and “RR” denote multi-scale technique [63] and random rotation augmentation. Unless otherwise specified, “6x” is used for HRSC and “1x” for the other datasets, while random flipping is the only augmentation that is always adopted by default.

DOTA-v1.0: Table 2 and Table 3 show that H2RBox-v2 outperforms HBox-Mask-Rbox methods in both accuracy and speed. Taking BoxLevelSet-RBox [22] as an example, H2RBox-v2 gives an accuracy of 15.02% higher, and a x7 speed faster by avoiding the time-consuming post-processing (i.e. minimum circumscribed rectangle operation). In particular, the recent foundation model for segmentation i.e. SAM [23] has shown strong zero-shot capabilities by training on the largest segmentation dataset to date. Thus, we use a trained horizontal FCOS detector to provide HBoxes into SAM as prompts, so that the corresponding masks can be generated by zero-shot, and finally the rotated RBoxes are obtained by performing the minimum circumscribed rectangle operation on the predicted Masks. Thanks to the powerful zero-shot capability, SAM-RBox based on ViT-B [69] in Table 2 has achieved 63.94%. However, it is also limited to the additional mask prediction step and the time-consuming post-processing, only 1.7 FPS during inference.

In comparison with the current state-of-the-art method H2RBox, to make it fair, we use the reproduced result of H2RBox, which achieves 70.05%, 2.23% higher than the original paper [20]. In this fair comparison, our method outperforms H2RBox by 2.26% (72.31% vs. 70.05%, both w/o MS). When MS is applied, the improvement is 2.62% (75.35% vs. 77.97%, both w/ MS).

Furthermore, the performance gap between our method and the RBox-supervised FCOS baseline is only 0.13% (w/o MS and RR) and 0.46% (w/ RR). When MS and RR are both applied, our method outperforms RBox-supervised FCOS by 0.57% (78.25% vs. 77.68%), proving that supplemented with symmetry-aware learning, the weakly-supervised learning can achieve performance on a par with the fully-supervised one upon the same backbone neural network. Finally, H2RBox-v2 obtains 80.61% on DOTA-v1.0 by further utilizing a stronger backbone.

DOTA-v1.5/2.0: As extended versions of DOTA-v1.0, these two datasets are more challenging, while the results present a similar trend. Still, H2RBox-v2 shows considerable advantages over H2RBox, with an improvement of 3.06% on DOTA-v1.5 and 1.65% on DOTA-v2.0. The results on DOTA-v1.5/2.0, HRSC, and FAIR1M are shown in Table 4.

HRSC: H2RBox [20] can hardly learn angle information from small datasets like HRSC, resulting in deficient performance. Contrarily, H2RBox-v2 is good at this kind of task, giving a performance comparable to fully-supervised methods. Compared to KCR [50] that uses transfer learning from RBox-supervised DOTA to HBox-supervised HRSC, our method, merely using HBox-supervised HRSC, outperforms KCR by 10.56% (89.66% vs. 79.10%).

FAIR1M: This dataset contains a large number of planes, vehicles, and courts, which are more perfectly symmetric than objects like bridges and harbors in DOTA. This may explain the observation that H2RBox-v2, learning from symmetry, outperforms H2RBox by a more considerable margin of 6.33% (42.27% vs. 35.94%). In this case, H2RBox-v2 even performs superior to the fully-supervised FCOS that H2RBox-v2 is based on by 1.02% (42.27% vs. 41.25%).

Loss in SS branch: Table 6 studies the impact of using the snap loss (see Sec. 3.4) and the angle coder. Column “PSC” indicates using PSC angle coder [36] and "w/o PSC" means the conv layer directly outputs the angle. Column “$\ell_{\text{s}}$” with check mark denotes using snap loss (otherwise using smooth L1 loss). Without these two modules handling boundary discontinuity, we empirically find that the loss could fluctuate in a wide range, even failure in convergence (see the much lower results in Table 6). In comparison, when both PSC and snap loss are used, the training is stable.

Loss in WS branch: Table 6 shows that CircumIoU loss with random rotation can further improve the performance, which H2RBox is incapable of. “$\ell_{\text{p}}$” means using CircumIoU loss in Sec. 3.4, and otherwise, IoU loss [61] is used following a conversion from RBox to HBox (see H2RBox [20]).

Weights between $L_{\text{flp}}$ and $L_{\text{rot}}$: Table 7 shows that on both DOTA and HRSC datasets, $\lambda=0.05$ could be the best choice under AP₅₀ metric, whereas $\lambda=0.1$ under AP₇₅. Hence in most experiments, we choose $\lambda=0.05$, except for Table 9 where $\lambda=0.1$ is used.

Range of view generation: When the rotation angle $\mathcal{R}$ is close to 0, the SS branch could fall into a sick state. This may explain the fluctuation of losses under the random rotation within $-\pi\sim\pi$, leading to training instability. According to Table 9, $\pi/4\sim 3\pi/4$ is more suitable.

Branch multiplexing: An additional experiment that randomly selects from 5% flip or 95% rotation in only one branch (the proportion based on $\lambda=0.05$ in Table 7) shows AP₅₀/AP₇₅: 72.24%/39.51% (DOTA w/o MS) while reducing the training time and the RAM usage to H2RBox’s level.

Padding strategies: Compared to the performance loss of more than 10% for H2RBox without reflection padding, Table 9 shows that H2RBox-v2 is less sensitive to black borders.

Annotation noise: Table 11 multiplies the height and width of annotated HBox by a noise from the uniform distribution $\left(1-\sigma,1+\sigma\right)$. When $\sigma=30\%$, the AP₅₀ of H2RBox-v2 drops by only 1.2%, less than H2RBox (2.69%), which demonstrates the better robustness of our method.

Training data volume: Table 11 displays that the gap between H2RBox and H2RBox-v2 becomes larger on the sampled version of DOTA, where $p$ denotes the sampling percentage.