Stable Diffusion For Aerial Object Detection

Aerial imagery [2], derived from various sources such as drones, satellites, and high-altitude platforms, plays a pivotal role in numerous real-world applications. One of the key tasks is object detection which aims to identify and locate objects of interest within a broader scene. However, the path to robust object detection in aerial images is challenging. Unlike ground-level images that are widely available, aerial data are restricted in dataset volume and generalization across different object classes [2]. An even more pressing concern is the long-tail distribution observed in these datasets, where certain classes of objects are vastly underrepresented. This sparsity of examples and the rare classes poses a significant hurdle for the training of generalized object detectors, as models often struggle to recognize and accurately detect these infrequent objects in real-world scenarios.

To mitigate the scarcity and long-tail distribution of aerial data, synthetic data has emerged as a promising solution [3, 4, 7]. Over the years, a plethora of synthetic techniques has been proposed, from the Copy-Paste method [3] where objects from existing images are superimposed onto different backgrounds to simulate new scenes, to advanced DNN-based image synthesis such as Generative Adversarial Networks (GANs) [4]. Recently, synthetic image generation has witnessed major advances from Diffusion-based techniques [7], which progressively improve an initial noisy image through a series of diffusion steps, and are able to generate photo-realistic natural or artistic images.

Although undeniably promising, applying diffusion-based synthetic methods to aerial object detection isn’t straightforward. Specifically, we observe two major challenges:

• •

  Sparse Region-of-Interests (ROIs): Diffusion methods have primarily been optimized for ground-level images abundant in semantic richness, while aerial detection often features object regions that are sparsely situated. For instance, while aerial urban scenes might be bustling with roads and buildings, aerial detectors might predominantly capture isolated cars or sparse ships traversing vast river expanses. This disparity in object density and semantics necessitates a tailored approach for diffusing aerial scenes.

• •

  Coordinate Extraction: While synthesizing whole images might be feasible with diffusion methods, a subsequent challenge arises for detection scenarios: how to extract the precise coordinates of the synthesized objects? Without this critical information, the synthetic images, no matter how realistic, may fall short in their utility for detection tasks.

To address these concerns, we introduce a novel framework tailored to the specificities of aerial images with the following steps:

• •

  Sparse-to-Dense ROI extraction: We extract semantically meaningful object patches from sparse aerial scenes, aiming to mirror the rich semantics typical of ground-level images. Paired with text prompts, diffusion models thus could be effectively trained to generate semantic meaningful object patches.

• •

  Fine-tuning using LORA: Training a foundation diffusion model from scratch is computing prohibitive. We thus leverage LORA (Low-Rank Adaptation) [5] to fine-tune the off-the-shelf stable diffusion model that circumvents the need for heavy full-model tuning.

• •

  Copy-Paste Composition: Finally, synthesized object patches are seamlessly integrated into real-world aerial background images using the Copy-Paste method [3], where coordinates can be easily obtained and facilitate the downstream object detection task.

Applied on the representative aerial object detection dataset DOTAv2.0 [2], our framework demonstrates consistent improvements on both overall and long-tailed classes, e.g., +1.2% to +2.7% mean mAP improvement, and at most +30.3% mAP improvement on extremely long-tail classes.

The current SD finetuning approach generates synthetic training images that brings new knowledge to detection model, as a result, mixing those images into original training dataset boosts detector mAP scores especially on long-tail classes. However we’ve also tried using only the synthetic images for training, without mixing in the original images from DOTAv2.0. We noticed the performance is extremely low, almost all classes got a close-to-zero mAP score. This may due to the fact that the synthetic images were composed without masks, detector may overfit to the boundries of the bboxes when trained only on those images. For the next step, we can try zero-shot segmentation methods to generate object masks, then compose synthetic images to eliminate current bbox boundries.

Aerial object detection task faces extensive dataset limitations like data collection constraints and skewed class distributions. Synthetic data, especially the recent diffusion-based techniques, emerged as a promising way to alleviate these constraints. Nevertheless, a direct application to aerial images is proved suboptimal, highlighting the need for a domain-specific approach. Our proposed framework, integrating sparse-to-dense semantic ROI extraction, LORA fine-tuning, and the Copy-Paste technique, presents a synergistic solution tailored to the unique challenges of aerial object detection. Extensive experiments demonstrate the effectiveness of our proposed synthetic framework, paving the way for more accurate and comprehensive aerial scene analyses.