Mapping Housing Stock Characteristics from Drone Images for Climate Resilience in the Caribbean

The Caribbean is among the world’s most climate-vulnerable regions due to the prevalence and intensity of extreme climate hazards such as storms, floods, and landslides. Category 5 hurricanes like Dorian, Irma, and Maria have devastated many small island developing states (SIDS) in recent years, leaving widespread trails of loss and destruction across the region. SIDS bear substantial economic costs from climate-extreme events, with the highest degree of damages often sustained in the housing sector [19]. Hurricane Maria, for example, destroyed over 90% of Dominica’s housing stock, accumulating costs over 200% of the nation’s GDP [18]. As global temperatures continue to rise, extreme weather events will only grow in severity, putting many more vulnerable homes and shelters at risk.

In response to these challenges, governments and international organizations have developed ambitious climate resilience programs to reduce the adverse effects of extreme climatic hazards in the housing sector [19, 9]. Climate resilience initiatives generally require comprehensive housing stock information to inform better retrofitting, reconstruction, and relocation plans. However, the traditional house-to-house approach to identifying high-risk buildings can be extremely expensive and time-consuming to implement, prompting the need for more timely and cost-efficient alternatives.

Recent years have seen a growing interest in using drones for disaster risk reduction and recovery [9, 28]. Previous works have successfully applied deep learning (DL) techniques to very high-resolution (VHR) aerial imagery to extract baseline housing information such as building footprints and rooftop attributes [27, 10, 22, 14, 24]. However, despite the evident advantages DL and Earth observation (EO)-based technologies, the widespread adoption of these solutions is often hindered by gaps in local capacity to develop and maintain systems for generating baseline exposure datasets.

This work aims to bridge these gaps by providing government agencies with an end-to-end workflow for rapidly generating critical baseline housing information using VHR drone images. Specifically, we leverage the Segment Anything Model (SAM) for building footprint delineation and convolutional neural networks (CNNs) for roof type and roof material classification. We also evaluate the cross-country generalizability of roof classification models across SIDS to determine the extent to which models trained in one country can be adapted to another. This work is developed under the Digital Earth Project for Resilient Housing and Infrastructure in partnership with the Government of the Commonwealth of Dominica (GoCD) and the Government of Saint Lucia (GoSL) to enhance the climate resilience of the housing sector of SIDS in the Caribbean.

To generate our ground truth datasets, we use (1) VHR aerial images in the form of aircraft- and drone-derived optical imagery and (2) building footprints in vector polygons, with Dominica and Saint Lucia as our primary regions of interest.

VHR Aerial Images. We acquired the following VHR aerial images from partner government agencies GoCD and GoSL, the World Bank Global Program for Resilient Housing (GPRH) [8], and the open data platform OpenAerialMap [5]: (1) aircraft-derived post-disaster orthophotos of Dominica; (2) aircraft-derived orthophotos of Saint Lucia; (3) pre- and post-disaster drone images of 10 villages and cities across Dominica; (4) drone images of 3 districts in Saint Lucia. The spatial resolutions of aircraft-derived images range from 10 to 20 cm/px, whereas the resolutions of drone-derived images range from 2 to 5 cm/px. For more information on the spatial resolution, coverage, and year of acquisition of the aerial images, see Table 3 in the Appendix.

Building Footprints Data. We obtained nationwide building footprints delineated from the aircraft-derived orthophotos of Dominica and Saint Lucia from the World Bank. For drone images with no corresponding building footprints, we initially looked to alternative data sources such as OpenStreetMap [6], Microsoft [4], and Google [2]. However, we found these publicly available building footprint datasets to be nonviable due to significant misalignment with the underlying drone images, as illustrated in Figure 2. To address this challenge, we used the Segment Anything Model (SAM) to delineate building instances directly from drone images [15]. Additional information on the SAM configuration used in this study is detailed in Section 3.

Rooftop Image Tiles. For roof classification, we selected approximately 15,000 building footprints across Dominica and Saint Lucia. For each building footprint, we extract the minimal bounding rectangle of the building polygon, scaled by a factor of 1.5, from the corresponding aerial image. We then proceeded to annotate a total of 9,150 buildings in Dominica and 5,175 buildings in Saint Lucia via visual interpretation of VHR aerial images. The buildings are classified based on two attributes: roof type and roof material, the class distributions of which are presented in Table 1. We note that Saint Lucia does not contain any samples for the "Blue Tarpaulin" category. Figure 1 illustrates examples of drone image tiles for each roof type and roof material category.

This section outlines our workflow for generating critical housing stock information in the Caribbean using drone images, as summarized in Figure 3.

To extract building footprints from drone images, we use Segment-Geospatial for segmenting raster images based on the Segment Anything Model (SAM) [29, 20, 21, 15]. We also leverage Language Segment Anything [16] to combine instance segmentation with text prompts to generate masks of specific objects in the drone images. We set our text prompt to "house", the box threshold (i.e. the threshold value used for object detection in the image) to 0.30, and the text threshold (i.e. the threshold value used to associate the detected objects with the provided text prompt) to 0.30. As a post-processing step, we apply the Douglas-Peucker algorithm to simplify the generated building polygons with a tolerance of $5e^{-6}$ [12].

Given the VHR aerial images and corresponding building footprint polygons, we proceed with developing our roof classification models. We begin by fine-tuning CNN models pre-trained on the ImageNet dataset [11] with architectures ResNet50 [13], VGG-16 [23], Inceptionv3 [25], and EfficientNet-B0 [26] using cross-entropy loss for both roof type and roof material classification tasks. The input rooftop image tiles are zero-padded to a square of size 224 x 224 px for ResNet50, EfficientNet-B0, and VGG-16 and 299 x 299 px for InceptionV3. We set the batch size to 32 and the maximum number of epochs to 60, and we use an Adam optimizer with an initial learning rate of $1e^{-}5$, which decays by a factor of 0.1 after every 7 epochs with no improvement. For data augmentation, we implement horizontal and vertical image flips with a probability of 0.50 and random rotations ranging from $-90^{\circ}$ to $90^{\circ}$. Given that our data is imbalanced (see Table 1), we implement random oversampling for the minority classes. To prevent overconfident predictions, we apply label smoothing as a regularization technique, with smoothing set to 0.1 [17].

For each country in our dataset, we set aside 80% of the data for training and the remaining 20% for testing, using stratified random sampling for the test set to preserve the percentage of samples per class as shown in Table 1. We note that the test sets for both countries are comprised entirely of drone images. Additionally, to test whether geographically diverse training data improves the prediction, we combine the training sets across Dominica and Saint Lucia (henceforth referred to as the "combined" dataset). We report the standard performance metrics F1-score, precision, recall, and accuracy.

Our results indicate that for Dominica, the best performance is achieved by an EfficientNet-B0 model for roof type classification (F1-score: 87.1%) and a ResNet50 model for roof material classification (F1-score: 89.5%). Likewise, for Saint Lucia, the best F1-score is attained by a ResNet50 model for roof type classification (F1-score: 89.5%) and an EfficientNet-B0 model for roof material classification (F1-score: 91.71%). For models trained on the combined dataset (i.e., "combined model"), the best results are obtained by an EfficientNet-B0 model for roof type classification (F1-score: 90.0%) and an Inceptionv3 model for roof material classification (F1-score: 90.4%). For the complete results and sample outputs, see Table 4 and Figure 4, respectively.

Next, we investigate the cross-country generalizability of the best-performing models by evaluating their performance on the designated test sets of each country. As shown in Table 2, our results indicate that for roof type classification, the combined model performs marginally better than models trained using only local data (i.e. data from the same country); however, for roof material classification, we find that locally trained models consistently outperform the combined model. In general, we find that local models demonstrate high levels of variability in performance in out-of-distribution countries, indicating the importance of collecting localized training data and the need for further studies on domain adaptation to reduce performance degradation in the face of geographical distribution shifts.

The overarching goal of this project is to strengthen local capacity in SIDS to leverage AI and EO-based solutions for resilient housing operations. To this end, our team is assisting government agencies in establishing Geographic Information Systems (GIS) units capable of generating, managing, and maintaining large-scale disaster risk datasets. We have partnered with the Humanitarian OpenStreetMap Team (HOT) [3] to design a training program on how to operate drones, coordinate pilots, process collected imagery, and manage the resulting geospatial datasets. Additionally, we have developed educational resources on geospatial data processing aimed at local government staff, community mappers, and disaster responders, including Google Colaboratory and Jupyter Notebook tutorials demonstrating how to run the SAM and CNN models to quickly generate baseline housing stock data from locally collected drone images [1, 7].

This work proposes a workflow for filling baseline exposure data gaps in the Caribbean using VHR drone images and DL. We demonstrate how computer vision and drone-based technologies can be used to rapidly generate housing stock information, including building footprints and roof classification maps, for disaster risk reduction and recovery. Based on our evaluation of the cross-country generalizability of DL models, we urge caution in applying locally-trained models off the shelf to new geographic regions and emphasize the importance of collecting local, highly contextualized training data. We also emphasize the importance of local capacity building, skills development, and co-creation of geospatial datasets in deploying sustainable AI-for-climate solutions, especially in Global South contexts. We conclude this study by urging governments in SIDS to invest in the digital infrastructure and local capabilities needed to sustainably generate and maintain EO-derived housing stock data for climate resilience.

This work is developed under Digital Earth for Resilient Housing and Infrastructure in the Caribbean, a World Bank project funded by the Global Facility for Disaster Reduction and Recovery (GFDRR), in partnership with the Government of the Commonwealth of Dominica (GoCD) and the Government of Saint Lucia (GoSL). This study builds on the initial work done by the World Bank’s Global Program for Resilient Housing (GPRH). We thank Mike Fedak and Christopher Williams for their assistance in providing data access.