Building Damage Mapping with Self-Positive Unlabeled Learning

Natural disasters, such as earthquakes, tsunami, and wildfires, are extremely rare occurrences that can wreak havoc on urban environments, resulting in enormous human and economic losses [19]. When a new natural disaster strikes, accurate and rapid actions are necessary in Humanitarian Assistance and Disaster Response (HADR) in order to save thousands of lives and prevent further damage [20]. Thus, a high-precision disaster damage mapping product needs to meet this demand. However, using very high-resolution optical and SAR images, the disaster damage areas are manually annotated by specialists, which might take a long time if the disaster area is huge [18].

Deep learning is an effective method to automatically recognize damages in remote sensing datasets  [2, 16]. However, deep learning models necessitate a huge number of training images as well as a large amount of high-quality labeled data  [8]. This is not possible in the case of new disasters. One possible solution is to use the datasets and labels of past disasters to identify the damages of new disasters. However, due to domain gaps caused by many factors, such as various geographical locations, different damage characteristics, and seasonal variation, models that have only been trained on previous disasters would not work well for future disasters.

Since human-expert labeling is time-consuming, only a small number of labels are used when annotations obtained after a disaster are used to train models. The availability of labeled data is extremely restricted right after a new disaster, yet an enormous amount of unlabeled satellite images can be acquired over the affected area. Lee et al. [14] applied two semi-supervised learning (SSL) techniques, MixMatch and FixMatch, for the building damages with very limited labeled samples (damaged and undamaged). However, it is very time-consuming to collect both damaged and undamaged samples. Positive-unlabeled learning (PU), which integrates only positive and unlabeled training data, has recently made significant progress [3]. By using PU techniques, we can train accurate damage assessment models for new disasters by spending less time on gathering fewer positive labels of damaged class.

In this work, the recent Self-PU learning [6] is applied to train building damage detection models using very limited labeled positive damage samples. The method is applied to three different datasets to validate the performance. We demonstrate that by utilizing only a fraction of the labeled positive damage, models may obtain outcomes that are comparable to those obtained from fully supervised methods.

Deep learning in building damage mapping Previous studies conducted in fully supervised settings have effectively utilized deep learning algorithms for building damage assessment. The xView2 Challenge [12] developed the first large-scale pre- and post-disaster datasets, namely xBD, for mapping building damage, which incorporates VHR optical imagery taken during 19 disasters, including floods, wildfires, and earthquakes. The winning solution adopted a Siamese neural network which is initialized by the building detection from pre-event dataset. Hao et al. [13] combined the features from pre- and post-disaster imagery with a Siamese attention-based Unet model. Gupta and Shah [11] proposed an end-to-end RescueNet for both building segmentation and individual building damage level assessment. Shen et al. [17] developed a cross-directional fusion strategy to investigate the relationships between the datasets taken before and after the disaster. Boin et al. [5] proposed the techniques to mitigate the imbalanced class problem for different damage levels. Adriano et al. [1] construct high-resolution optical imagery and high-to-moderate-resolution SAR data. The multi- and cross-modality deep neural networks are applied to the new dataset.

In the above studies, the testing datasets are drawn from the same distribution as the training dataset. They do not address the difficulty of conducting inference for a disaster that just happened with very limited labeled datasets. Xu et al. [22] and Benson and Ecker [4] investigated the generalization experiments among the disasters in different regions and indicated that a significant generalization gap existed in current state-of-the-art models. The findings of models trained on previous disasters did not perform well when applied to a new disaster that did not have any labeled datasets.

Positive-unlabeled (PU) learning PU learning is a subdivision of SSL. However, when compared to SSL, there are no negative samples in PU learning, making the task more difficult and necessitating the invention of an algorithm to compensate for the lack of negative samples [3]. There are currently two solutions: 1) find credible negative samples in unlabeled data using heuristics or semi-supervised learning approaches [15], 2) treat unlabeled samples as weighted negative ones, including non-negative PU (nnPU) [7]. To further improve the performance of PU learning, the self-paced learning [6], active learning is considered in the framework of PU learning. In the remote sensing applications, Gui et al. [10] investigated eigenvalue statistical components-based PU-Learning to extract built-up from PolSAR datasets. Wu et al. [21] produced a landslide susceptibility map via the PU-bagging technique.

ABCD. ABCD (AIST Building Change Detection) dataset ¹¹1https://github.com/gistairc/ABCDdataset [9] is a new labeled dataset to identify whether buildings have been washed-away by tsunami of Tohoku region of Japan in 2011. The class label assigned to each patch pair (i.e. "damaged" or "survived") represents whether or not a building at the center of the before-tsunami patch got washed-away by tsunami. In this work, the number of images (the size is $160\times 160$) for "damaged" and "survived" are 4253/4253.

xBD. In additional to ABCD dataset for 2011 Tohoku earthquake, we generated the patch-based datasets for the 2018 Palu tsunami and 2018 Hurricane Michael from xBD. There are four damage levels, including no damage, minor damage, major damage and destroyed. To formulate the binary classification problem and make the class label same as ABCD, major damage and destroyed are categorized into the class Damaged, while no damage, and minor damage are combined as the class Survived. The patches of each building were cropped from the original WorldView images of xBD with a fixed size of 128 $\times$ 128 pixels. Thus, the total number of samples for the 2018 Palu tsunami and 2018 Hurricane Michael are 7645 (27% as the damaged) and 9214 (29% as the damaged), respectively.

The objective is formulated as a binary classification with two classes ($\bm{X}\in\mathcal{R}^{6}$, $Y\in\{-1,+1\}$): damaged and survived, where $\bm{X}$ is the input that contains six bands of pre- and post-event RGB and $Y$ is the class label. The damaged and survived classes are treated as the positive ($+1$) and negative ($-1$). In the disaster damage mapping, we only have few numbers of damaged samples (P). The remaining datasets are unlabeled (U), which include positive or negative. Assume that we have the training dataset $D$, which include positive ($D_{P}$) and unlabeled datasets ($D_{U}$). Then, we have $D=D_{P}+D_{U}$, where $D_{P}$ and $D_{U}$ contain $n_{P}$ and $n_{U}$ samples, respectively. Let $p(x,y)$ be the underlying joint distribution of $(\bm{X},Y)$, $p(x|Y=+1)$ and $p(x|Y=-1)$ are the distributions of P and N datasets. $\pi_{p}=p(Y=+1)$ and $\pi_{n}=p(Y=-1)=1-\pi_{p}$ are the positive and negative class-prior probabilities. As shown in Fig. 1, we firstly train a classification model to create a trusted pseudo PN dataset for U ($D_{trust}$), and the remaining is treated as a new PU dataset ($D_{U}-D_{trust}$). Then, knowledge distillation with two student models and their teacher models is used as a regularization to enhance performance while overcoming the inconsistent between the two models.

Due to the fact that deep neural networks are prone to memorizing simple patterns first, followed by irregular patterns, We might start by looking at simple instances and labeling them with confidence. Given an input $x$ with the label $y$ to the given model $g$, the probability of $x$ being positive are calculated. A larger probability implies greater confidence that $x$ belongs to the positive class, and vice versa. Based on this assumption, we selected $n$ most confident positive and negative samples from $D_{U}$ to form $D_{trust}$. Then, the loss function that integrates the supervised one of $D_{trust}$ and PU loss of $D_{U}-D_{trust}$, is given by

$$L_{hybrid}=\sum_{x\in D_{trust}}L_{CE}(x,y)+\sum_{x\in D_{U}-D_{trust}}L_{nnPU}(x)$$

(1)

where, the cross-entropy loss are used for $D_{trust}$ and non-negative PU (nnPU) [7] are used for $D_{U}-D_{trust}$. To make the the selected confident samples as accurate as possible, We start by training 10 epochs to “warm up” the model. Two ways are used to select the trusted pseudo PN datasets: 1) fixed size: the model choose the fixed number of samples while removing lower-confidence samples, 2) without replacement: the model choose a fixed number of samples without removing lower-confidence samples. In this work, we adapt the similar dynamic selection strategy [6]: linearly increase the size of $D_{trust}$ from 0 to 20% of $D_{u}$ with the 75% bootstrap size of training set for the model.

Then, knowledge distillation with two student models ($g_{1}$ and $g_{2}$) and two teacher models ($G_{1}$ and $G_{2}$) are further applied [6]. The two student networks shared the same $D=D_{P}+D_{U}$ and same networks architectures. However, two different trusted pseudo PN datasets may be extracted by two student networks, thus, the consistently between two student networks should be considered as the distillation via MSE loss. The distillations from $g_{1}$ to $g_{2}$ and from $g_{2}$ to $g_{1}$ are given by:

$$L_{MSE}(g_{1},g_{2},x)=\|f(g_{1}(x))-f(g_{2}(x))\|^{2},x\in D-D_{trust1}$$

(2)

$$L_{MSE}(g_{2},g_{1},x)=\|f(g_{2}(x))-f(g_{1}(x))\|^{2},x\in D-D_{trust2}$$

(3)

where, $f$ is a monotonic function of mapping (e.g., sigmoid function). The loss function of the student networks is given by (we only consider the MSE over hard unlabeled samples with larger nnPU):

$$L_{student}=\sum_{x\in D_{U}-D_{trust1}}L_{S}(g_{1},g_{2},x)+\sum_{x\in D_{U}-D_{trust2}}L_{S}(g_{2},g_{1},x)$$

(4)

where, $L_{S}(g_{1},g_{2},x)=L_{MSE}(g_{1},g_{2},x)$ when $L_{nnPU}(x)>\alpha L_{MSE}(g_{1},g_{2},x)$, otherwise $L_{S}(g_{1},g_{2},x)=0$, $\alpha$ is the control parameter.

Beside the distillation between two student networks, two teacher networks are included to make sure the consistency with the weighted moving average trajectory of the student networks. The weights of $G_{1}$ and $G_{2}$ are updated as

$$\Theta_{1,t}=\theta_{1,t-1}+\beta\theta_{1,t}$$

(5)

$$\Theta_{2,t}=\theta_{2,t-1}+\beta\theta_{2,t}$$

(6)

where, $\Theta_{1,t}$ and $\theta_{1,t}$ are the parameters of $G_{1}$ and $g_{1}$ at time $t$, respectively. $\beta$ is the user-defined parameter. The distillations from $G_{1}$ to $g_{1}$ and from $G_{2}$ to $g_{2}$ are given by:

$$L_{teacher}=\sum_{x\in D_{U}}\|f(G_{1}(x))-f(g_{1}(x))\|^{2}+\sum_{x\in D_{U}}\|f(G_{2}(x))-f(g_{2}(x))\|^{2}$$

(7)

The total loss function is given by:

$$L=L_{hybrid}+L_{student}+L_{teacher}$$

(8)

Settings. The efficacy of the proposed methods on classifiy damages are tested on ABCD and xView2 datasets. We split the datasets with the ratios of 4/3/3 for train/validation/test. To formulate the PU setting, we select r% (r ranges from 10 to 30) damaged samples as P, and the remaining (1-r)% damaged and 100% survived class are treated as U. The fully-supervised learning with the following setting: 1) Full PN; 2) Standard PN (r% P+100% N); 3) Small PN (r% P+r% N); 4) PU (U is treated as the N) and nnPU are used as the baselines. In this work, we adopt the simple VGG11 that contains eight convolutional layers and three full-connected layer. The pre- and post-disaster RGB images are treated as the input of the models. We set the SGD with momentum of 0.9 and a batch size of 16 for 100 epochs on a single NVIDIA V100 GPU with 16GB memory.

Results. Table 1 reports the accuracies of different methods. The fully-supervised model on all labeled data (Full PN), which is not available in real scenario, can be treated as the upper bound of the performance. The fully-supervised model with PU setting obtained the worst results because the training label is very noisy, especially for the negative labels. The clean training sets with small amount of P (standard PN and small PN) improved the results. nnPU makes further improvements in all cases. Self-PU improves the performance on all three disaster datasets. It should be noticed that we can obtain nearly the accuracies of 90% using only 30% positive damaged class as the input.

Ablation studies. From Table 2, it can be clearly seen that the proposed Self-PU methods gain the better performance than the ones which only consider the $L_{hybird}$ and $L_{hybird}+L_{student}$. Table 3 investigates three different values of $\alpha$ and $\beta$. $\alpha$ indicates the quality of consistency between two students and $\beta$ controls the updates of teachers from the students. Both parameters favors a reasonably moderate values. Larger or smaller values will decrease the accuracies.

In this paper, we introduced a novel application of PU learning to detect damaged buildings in satellite imagery with limited labeled samples. We experimented with the recent PU techniques with the combination of self-paced learning and knowledge distillation, and showed how they are able to achieve strong performance using very limited samples of damaged buildings by leveraging unlabeled samples. They consistently outperformed fully supervised models and even achieved performance close to that of a fully supervised setting with no data constraints. The results empirically showed how self-PU learning approaches can be useful to train models when a new disaster is unfolding in an unseen region. For future work, we will work towards the combination of the labeled datasets from past disasters and very limited damaged samples to further improve the performance. .

In the HADR domain, the most urgent need is to provide the building damage mapping that requires a large number of labeled datasets as the input of current SOTA deep learning models, which are already explored in the past AI for HADR workshops [17, 5]. However, such labeled datasets are not available for a new disaster. Previous studies in [22] and [4] proved that the result of a new disaster is not accurate by training the model on the past disasters. Lee et al. [14] attempted to apply SSL methods for the building damage mapping. In our work, we provided the building damage mapping using very limited positive damaged images, requiring less time for the annotation before the response.

This work was supported by KAKENHI (Grant Number 19K20309 and 19H02408), JSPS Bilateral Joint Research Project (Grant number JPJSBP 120203211).