Transformation-Invariant Network for Few-Shot Object Detection in Remote Sensing Images

Given a query image $I^{q}\in\mathbb{R}^{C\times H\times W}$, we first generate its transformed version $I^{q}_{t}\in\mathbb{R}^{C\times H\times W}$. Subsequently, both $I^{q}$ and $I^{q}_{t}$ are passed through a shared backbone network and Feature Pyramid Network (FPN) to produce feature maps $f^{q,b}$ and $f^{q,b}_{t}$, respectively. To ensure consistency in the generated proposals, only $f^{q,b}$ is utilized as input for the Region Proposal Network (RPN). The same transformation is applied to generate transformed proposals for $I^{q}_{t}$. Finally, RoI features $f^{q}\in\mathbb{R}^{C_{q}\times H_{q}\times W_{q}}$ and $f^{q}_{t}\in\mathbb{R}^{C_{q}\times H_{q}\times W_{q}}$ are extracted using the RoI Align operation, with $H_{q}$ and $W_{q}$ set to 7.

Similar to previous works [22, 24], the support branch takes 4-channel N-way K-shot support images $I_{i}^{s}$ as input, consisting of an RGB image and its binary mask derived from the object bounding box. The support feature maps $f^{s}\in\mathbb{R}^{C_{s}\times 1\times 1}$ are obtained through backbone feature extraction and global average pooling (GAP) operation.

For feature aggregation, we employ a 1 $\times$ 1 depth-wise convolution, which is both simple and effective. Specifically, the support feature $f^{s}\in\mathbb{R}^{C_{s}\times H_{s}\times W_{s}}$ serves as a convolution kernel for a 1 $\times$ 1 depth-wise convolution. The resulting aggregated features, denoted as $f^{agg}$ and $f^{agg}_{t}$, are then fed into the RoI Head to obtain classification scores $\theta^{cls}$, $\theta^{cls}_{t}$, and regression parameters $\theta^{reg}$, $\theta^{reg}_{t}$.

By minimizing the consistency loss, the network can perform consistency detection results on both query image processing and its transformation version so that the aggregation feature $f^{agg}$ and $f^{agg}_{t}$ are forced to the same distribution. The consistency loss comprises two components: the classification consistency loss $L_{cls-c}$ and the regression consistency loss $L_{reg-c}$. Let $\sigma^{j}$ and $\sigma^{j}_{t}$ denote the classification distribution for the $j$-th proposal in $\theta^{cls}$ and $\theta^{cls}_{t}$, respectively. Different metrics, such as $L_{2}$ loss, Jensen‒Shannon divergence (JSD), and Kullback‒Leibler divergence (KLD), can be used to measure the distance between these distributions. Through experiments, we find that $L_{2}$ loss performs the best (as discussed in Section IV-D3). Therefore, the classification consistency loss can be defined as follows:

In contrast to the classification distribution, regression parameters change with image transformation. Let $[\Delta x^{j},\Delta y^{j},\Delta w^{j},\Delta h^{j}]$ represent the regression results for the $j$-th proposal in $\theta^{reg}$, representing the offset of the center point and the scale coefficients for width and height. Similarly, $[\Delta x^{j}_{t},\Delta y^{j}_{t},\Delta w^{j}_{t},\Delta h^{j}_{t}]$ represents the $j$-th parameters in $\theta^{reg}_{t}$. In this paper, we focus on the effect of three flipping transformations: horizontal flipping, vertical flipping, and diagonal flipping. Since the flipping transformation causes $\Delta x^{j}_{t}$ or $\Delta y^{j}_{t}$ to move in the opposite direction, a negative operation is applied to correct it. For example, in diagonal flipping, $\Delta x^{j}$ and $\Delta y^{j}$ correspond to $-\Delta x^{j}_{t}$ and $-\Delta y^{j}_{t}$. The regression consistency loss with $L_{2}$ regularization can be defined as:

For the other two flipping transformations, $\Delta x^{j}$ and $\Delta y^{j}$ should correspond to $-\Delta x^{j}_{t}$ and $\Delta y^{j}_{t}$ for horizontal flipping and $\Delta x^{j}_{t}$ and $-\Delta y^{j}_{t}$ for vertical flipping. The complete procedure to compute the consistency loss is presented in Algorithm 1. Both the base training phase and the few-shot fine-tuning phase utilize the consistency loss.

We optimize the total loss $L_{T}$ by combining the standard supervised loss terms with the consistency losses, as follows:

Here, $L_{rpn}$, $L_{cls}$, and $L_{reg}$ represent the losses (cross-entropy loss and L1 loss) for the Region Proposal Network (RPN), the classification loss and regression loss, respectively. The terms $\lambda_{1}$ and $\lambda_{2}$ control the strength of the transformation consistency. As object detection consistency is not stable in the early training stage, we choose relatively smaller weights: $\lambda_{1}=0.05$ and $\lambda_{2}=0.02$.

During training, the support feature is randomly selected for each iteration. However, during testing, all support features must be involved in the process. The testing phase is outlined in Algorithm 2. Importantly, the transformed images are not included in the testing phase, ensuring that there is no impact on the overall inference time. The impact of different components in the consistency loss on both training and testing time is discussed in Section IV-D6.

We present the results of different methods for split1 and split2 in Tab. III and Tab. IV respectively. In split1, as shown in Tab. III, in addition to the reproduced generic few-shot object detection methods, we further incorporate comparisons with RepMet [50] with InceptionV3 [55] as the backbone and FSRW with DarkNet-19 [56] as backbone according to [33]. We make the following observations from the results. First, the TINet outperforms all competing methods except for nAP@5shot. The gap is particularly significant compared with generic FSOD methods where more than $10\%$ improvements in nAP/bAP/mAP are observed throughout 5 to 30 shots. This suggests the strong few-shot learning capability of TINet. Second, all the transfer-learning approaches and meta-learning approaches outperform the original fine-tuned Faster-RCNN (FRCN-ft). For the meta-learning approaches, we observe that Meta-RCNN and FsDetView only use the C4 layer for subsequent processing, thus they perform relatively worse compared with the Strong Baseline on the DIOR dataset which features large diversity in object scales. For the transfer-learning-based approaches, both FSCE and TFA are way behind TINet despite all are using FPN, probably due to the large intra-class variation in the DIOR dataset. Finally, compared with the results reported by the state-of-the-art methods in RSIs [33][37], we also demonstrate a competitive result, except for the nAP@5 shots. We further present the comparisons for split2 in Tab. IV, which is generally considered to be more challenging than split1. Under split2, we observe more significant improvements in TINet from the best-performing methods. These results again validate the effectiveness of the proposed method.

The quantitative results obtained by applying different methods to the HRRSD dataset are presented in Table V. Since no previous algorithms have been tested on the HRRSD dataset, we can only compare the results with our implemented algorithm. From the results, we can observe that most methods achieve good performance on this dataset, which is less complex compared to the DIOR dataset and has fewer object categories. Transfer learning approaches, especially FSCE, demonstrate strong performance in certain aspects. However, our method still outperforms FSCE in terms of nAP at 5 shots, 10 shots, 20 shots, and 30 shots, with improvements of 0.8%, 1.0%, 4.4%, and 2.4%, respectively.

As shown in Table VI, since the NWPU VHR-10.v2 dataset is relatively simple, all the methods achieve relatively good results. It can be observed that our Strong Baseline has higher results than the general meta-learning methods but lower results than the transfer learning approach, FSCE. This is because this dataset is very small and the intra-class similarity is not large. TINet still outperforms FSCE on most metrics. Although OFA [39] improves object recognition in novel categories by rotating the support samples, increases the inference time, and does not use the oriented feature augmentation method in the base training phase, which may reduce the generalization performance.

There are two consistency losses ($L_{cls-c}$ and $L_{reg-c}$) in the TINet. As shown in Tab. VII, we examine the influence of $L_{cls-c}$ and $L_{reg-c}$) and make a comparison with data augmentation, which augments the image before feeding it into the query branch. The experimental results in the first row are obtained without any strategy. It should be noted that for data augmentation, we choose a combination of horizontal, vertical, and diagonal flipping. The consistency loss $L_{cls-c}$ and $L_{reg-c}$ here are both the $L_{2}$ loss, and the corresponding flipping method is diagonal flipping. It can be observed that data augmentation improves the performance of the network. However, as the number of shots increases, the impact of data augmentation diminishes significantly. On the other hand, adding only the consistency losses $L_{cls-c}$ and $L_{reg-c}$ outperforms the use of data augmentation alone. When either $L_{cls-c}$ or $L_{reg-c}$ is added, the network achieves stable improvements in performance, except in the 5-shot scenario. The best results are obtained when both the consistency losses and data augmentation are used together, indicating that these two techniques are complementary to each other.

We verify the effect of the different flipping transformations on the experimental results. As shown in Tab. VIII, we observe similar results for both the horizontal and vertical flips. The result of the diagonal flip is slightly better than the previous two. This is because the diagonal flip introduces fewer changes to the object’s appearance so that the training is more stable compared to the horizontal and vertical flips.

We verify the effect of different regularizations in the classification consistency loss $L_{con-c}$ on the results. As shown in Tab. IX, the JSD and KLD represents Jensen–Shannon divergence and Kullback–Leibler divergence, respectively. The weight of JSD and KLD here we chose are 0.05 and 0.1. It can be observed that simply using the $L_{2}$ loss yields the best results in most metrics except in 5-shot. Because the $L_{2}$ loss is more sensitive to outliers, it can play a more restrictive role.

General object detectors always focus on two main sub-tasks (regression and classification) so that the weight of auxiliary losses should be relatively smaller. We carried out evaluations on the robustness of the choice of $\lambda$s in Tab. X. In general, stable performance is observed around the hyper-parameters we chose.

We further measure the calibration of detection models by the Pearson Correlation Coefficient (PCC), defined as follows:

$x$ and $y$ are respectively the IoU between ground-truth and predicted bounding boxes and the confidence score (the highest posterior). A high correlation indicates the confidence is well calibrated. $m_{x}$ and $m_{y}$ are the mean of the $x$ and $y$ respectively. The results (shown in Fig. 4) demonstrate that the TINet obtained a higher value of PCC than the Strong Baseline and Meta-RCNN in all the datasets, suggesting TINet is better calibrated than others.

The results are shown in Tab. XI. Here one iteration includes four multi-scale images, and in the testing phase, the images are resized to 600 $\times$ 600. It can be observed that the training time of TINet increases slightly compared with Strong Baseline because TINet has to process two images simultaneously. However, the inference time of all the comparison methods remains the same. In addition, the loss computation has little effect on the time of network training.

We considered this approach, but experimental results showed no significant difference in the results compared to the current method (shown in Fig. XII). Therefore, to ensure training efficiency, we only apply transformations in the query branch. The possible reason for this is that by transforming instances in the query branch while keeping the support instances fixed, the model can learn a sufficient variety of matching methods. In this scenario, adding these transformations in the support branch became unnecessary.

Initially, we utilized the meta-loss but later discovered that the results were not better than without using the meta-loss (shown in Tab. XIII). We hypothesize that the query feature contains both regression and classification information, while the support feature only contains classification information. In our case, although the meta-loss can improve the classification performance of the support feature, it may not necessarily be beneficial for the regression task.

Due to the limited training samples in FSOD, objects near the edges of the image can be lost when dealing with objects rotated arbitrarily (see Fig. 5). Therefore, we did not include arbitrary rotations in the transformations. Other geometric transformations might enhance the model’s performance. We will validate this in our future work.

The sensitivity of FSOD to the selected support samples is also crucial. For results in Tab. XIV, we carried out 10 runs with different support samples for FSCE (second best model) and our method (TINet). We observe from Tab. XIV that TINet is slightly better than FSCE and the results are relatively stable w.r.t the choice of support samples.

We finally compared our method with two representative transformation invariant methods. ReResNet [45] achieves invariance by extracting rotation-invariant features, while TIP introduces Cutout and Gaussian Noise into the input images and utilizes consistency loss for achieving invariance. From Tab. XV, we can observe that the performance of the Strong Baseline deteriorates significantly after incorporating ReResNet. This is because ReResNet lacks a sufficient number of samples for training in a few-shot setting, leading to convergence issues. Moreover, the inference speed is significantly slower when using ReResNet (4.4 FPS compared to 15.5 FPS). As TIP [41] did not publish their source code, we managed to replace our geometric transformation with Cutout and Gaussian Noise, as proposed in TIP. The results suggest that geometric transformation is significantly superior to Cutout and Gaussian Noise, especially in the low-shot regime.

We generated a confusion matrix using the detection results of the NWPU VHR-10.v2 test set. The abbreviations for the categories are as follows: AP-airplane, BD-baseball diamond, BC-basketball court, BR-bridge, GTF-ground track field, HA-harbour, SH-ship, ST-storage tank, TC-tennis court, VE-vehicle, and BG-background. Unlike the classification task, the detection task involves cases of false positives and missing detections, making the inclusion of a background class necessary to cover all cases. It should be noted that the percentage sum along the horizontal axis is 100%, while the vertical axis is not 100% due to normalization based on the horizontal axis. From Fig. LABEL:fig:cm_comparison, it can be observed that the Strong Baseline has a low probability of correctly recognizing the novel classes (AP, BD, and TC). Our method, on the other hand, alleviates this problem to some extent. Additionally, it is worth mentioning that our method does not forget the characteristics of the base class while training on the novel class.

As shown in Fig. 7, we present a comparison of several FSOD methods on the DIOR dataset of 30-shot at split1, which contains objects of the novel category, including airplanes, baseball fields, train stations, tennis courts, and windmills, as well as a small number of objects from the base category. The results highlight the strong generalization ability of our proposed method, TINet, attributed to its multi-scale feature structure and transformation invariant learning. Notably, we observe that for smaller objects like airplanes and windmills, Meta-RCNN without the FPN structure performs even worse than the original fine-tuning Faster-RCNN (FRCN-ft). However, the incorporation of the Strong Baseline significantly enhances the detection performance of Meta-RCNN, leading to similar results as FSCE. Moreover, leveraging the transformation invariant strategy atop the Strong Baseline, TINet further improves the detection performance for objects with varying orientations, such as airplanes and tennis courts. For simpler objects like baseball fields, which lack scale and orientation diversity, all the compared algorithms achieve comparable detection results. Overall, our TINet outperforms all competing methods, producing the best detection results.

Furthermore, we provide additional qualitative results in Fig. 8, encompassing both novel and base objects on the HRRSD (30-shot) dataset. This dataset exhibits less complexity compared to DIOR, resulting in fewer false detections. For example, the appearance of the circular aircraft waiting hall is very similar to the storage tank, so it caused false detection. The color of the missing airplane is overlaid with more red, which makes it different from other airplanes. Hence, this phenomenon motivates us should focus on designing modules to extract more discriminative features in future work.